diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index bb0e323c943..17edb0b91af 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -27,7 +27,9 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima * Fixed `gremlin-go` to report a malformed or truncated GraphBinary response as a deserialization error rather than a bare decoder message. * Made `TinkerGraph` an interface and renamed the in-memory implementation to `TinkerMemoryGraph`; `TinkerGraph.open()` and `gremlin.graph=...TinkerGraph` behave as before. *(breaking)* -* Renamed `TinkerTransactionGraph` to `TinkerStorageGraph`. *(breaking)* +* Renamed `TinkerTransactionGraph` to `TinkerStorageGraph`. +* Added a pluggable storage layer to `TinkerStorageGraph` that durably persists each committed transaction to disk. +* Removed automatic persistence from `TinkerMemoryGraph`, which is now purely in-memory and ignores `gremlin.tinkergraph.graphLocation`/`graphFormat`. Use `TinkerStorageGraph` for durability or `g.io()` for interchange. [[release-4-0-0-beta-3]] === TinkerPop 4.0.0-beta.3 (July 20, 2026) diff --git a/docs/src/reference/implementations-tinkergraph.asciidoc b/docs/src/reference/implementations-tinkergraph.asciidoc index 4e3c873e3c0..6795c5a0c7a 100644 --- a/docs/src/reference/implementations-tinkergraph.asciidoc +++ b/docs/src/reference/implementations-tinkergraph.asciidoc @@ -39,13 +39,15 @@ under the License. ---- -image:tinkerpop-character.png[width=100,float=left] TinkerGraph is a single machine, in-memory (with optional -persistence), graph engine that provides both OLTP and OLAP functionality. It is non-transactional by default but does +image:tinkerpop-character.png[width=100,float=left] TinkerGraph is a single machine, in-memory graph engine that +provides both OLTP and OLAP functionality. It is non-transactional by default but does have a lightweight transactional form that can be instantiated offering simple `ThreadLocal` transactions supporting -`read committed` transaction isolation. As of 4.0.0, `TinkerGraph` is an interface with two implementations: -`TinkerMemoryGraph`, the in-memory, non-transactional implementation that `TinkerGraph.open()` constructs, and -`TinkerStorageGraph`, the transactional implementation formerly named `TinkerTransactionGraph`. TinkerGraph is -deployed with TinkerPop and serves as the reference +`read committed` transaction isolation, with optional durable persistence to disk. As of 4.0.0, `TinkerGraph` is an +interface with two implementations: `TinkerMemoryGraph`, the in-memory, non-transactional implementation that +`TinkerGraph.open()` constructs, and `TinkerStorageGraph`, the transactional implementation formerly named +`TinkerTransactionGraph`. Transactions and durability are separate concerns. `TinkerStorageGraph` writes nothing to +disk unless a storage engine is configured, and without one it is a purely in-memory graph that retains its +transaction support. TinkerGraph is deployed with TinkerPop and serves as the reference implementation for other providers to study in order to understand the semantics of the various methods of the TinkerPop API. Its status as a reference implementation does not however imply that it is not suitable for production. TinkerGraph has many practical use cases in production applications and their development. Some examples of TinkerGraph @@ -172,19 +174,28 @@ TinkerGraph has several settings that can be provided on creation via `Configura |gremlin.tinkergraph.defaultVertexPropertyCardinality |The default `VertexProperty.Cardinality` to use when `Vertex.property(k,v)` is called. |gremlin.tinkergraph.allowNullPropertyValues |A boolean value that determines whether or not `null` property values are allowed and defaults to `false`. |gremlin.tinkergraph.vertexLabelCardinality |The `LabelCardinality` for vertices. Options are `ONE` (default, single immutable label, backward-compatible with 3.x), `ONE_OR_MORE` (at least one label, mutable), or `ZERO_OR_MORE` (zero or more labels). See <>. -|gremlin.tinkergraph.graphLocation |The path and file name for where TinkerGraph should persist the graph data. If a -value is specified here, the `gremlin.tinkergraph.graphFormat` should also be specified. If this value is not -included (default), then the graph will stay in-memory and not be loaded/persisted to disk. -|gremlin.tinkergraph.graphFormat |The format to use to serialize the graph which may be one of the following: -`graphml`, `graphson`, `gryo`, or a fully qualified class name that implements Io.Builder interface (which allows for -external third party graph reader/writer formats to be used for persistence). -If a value is specified here, then the `gremlin.tinkergraph.graphLocation` should -also be specified. If this value is not included (default), then the graph will stay in-memory and not be -loaded/persisted to disk. +|gremlin.tinkergraph.storage |The durable storage engine used by `TinkerStorageGraph` to persist committed transactions +to disk. The value is either a built-in engine name (`graphbinary`) or a fully qualified class name of a +`TinkerStorage` implementation. When not specified (default), the graph holds data only in memory. This setting is +only valid on `TinkerStorageGraph` and is ignored by the in-memory `TinkerMemoryGraph`. +|gremlin.tinkergraph.storage.directory |The directory in which `TinkerStorageGraph` stores its durable data. Required +when `gremlin.tinkergraph.storage` is set and ignored otherwise. +|gremlin.tinkergraph.storage.sync |The durability applied to each committed transaction by a `TinkerStorageGraph` +storage engine. `commit` (default) forces each commit to disk so an acknowledged commit survives an operating system +crash or power loss. `os` only flushes to the operating system, so a commit survives a crash of the JVM process but +may be lost on an operating system crash or power loss. Only meaningful when `gremlin.tinkergraph.storage` is set. +|gremlin.tinkergraph.storage.compactThreshold |The size in bytes at which a `TinkerStorageGraph` storage engine +automatically compacts its append log, bounding the log growth and restart time of a long-running graph that is never +explicitly closed. Defaults to `67108864` (64 MB). A value of `0` disables automatic compaction, leaving it to +`close()` or an explicit `compact()`. Only meaningful when `gremlin.tinkergraph.storage` is set. +|gremlin.tinkergraph.storage.preserveVertexPropertyIds |Whether a `TinkerStorageGraph` storage engine persists +auto-generated vertex-property ids so they are stable across a close and reopen. Defaults to `false`, which +regenerates those ids on load and keeps the store smaller. Element and edge ids are always preserved regardless. Only +meaningful when `gremlin.tinkergraph.storage` is set. |========================================================= -NOTE: To use <>, configure `gremlin.graph` as -`org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph`. +NOTE: To use <> and <>, configure +`gremlin.graph` as `org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph`. The `IdManager` settings above refer to how TinkerGraph will control identifiers for vertices, edges and vertex properties. There are several options for each of these settings: `ANY`, `LONG`, `INTEGER`, `UUID`, `STRING` or the @@ -197,32 +208,11 @@ type as well as generate new identifiers with that specified type. TIP: Setting the `IdManager` to `ANY` also allows `String` type ID values to be used. -If the TinkerGraph is configured for persistence with `gremlin.tinkergraph.graphLocation` and -`gremlin.tinkergraph.graphFormat`, then the graph will be written to the specified location with the specified -format when `Graph.close()` is called. In addition, if these settings are present, TinkerGraph will attempt to -load the graph from the specified location. - -IMPORTANT: If choosing `graphson` as the `gremlin.tinkergraph.graphFormat` with untyped GraphSON, be sure to also -establish the various `IdManager` settings to ensure that identifiers are properly coerced to the appropriate types, -as untyped GraphSON can lose the identifier's type during serialization (i.e. it will assume `Integer` when the -default for TinkerGraph is `Long`, which could lead to load errors that result in a message like, "Vertex with id -already exists"). Typed GraphSON preserves the identifier's type across the serialization round-trip (for example, a -`Long` identifier is written as `{"@type":"g:Int64","@value":...}`), so the `IdManager` coercion above is not -required in that case. - -These settings can be supplied programmatically by building a `Configuration` and passing it to `TinkerGraph.open()`. -For example, to persist to disk as GraphSON while keeping `Long` identifiers: - -[source,java] ----- -BaseConfiguration conf = new BaseConfiguration(); -conf.setProperty("gremlin.tinkergraph.graphLocation", "/tmp/tinkergraph.json"); -conf.setProperty("gremlin.tinkergraph.graphFormat", "graphson"); -conf.setProperty("gremlin.tinkergraph.vertexIdManager", "LONG"); -Graph graph = TinkerGraph.open(conf); -// ... work with the graph ... -graph.close(); // persists the graph to the configured graphLocation ----- +Durable persistence is provided by `TinkerStorageGraph` through its pluggable storage layer and is described in the +<>. It is enabled by configuration rather than by the choice of +implementation. Neither `TinkerMemoryGraph` nor a `TinkerStorageGraph` without a configured storage engine holds data +across JVM restarts. Moving data in and out of any TinkerGraph in an interchange format is handled on demand by the +`io()` step rather than by graph configuration, as shown below. It is important to consider the data being imported to TinkerGraph with respect to `defaultVertexPropertyCardinality` setting. For example, if a `.gryo` file is known to contain multi-property data, be sure to set the default @@ -278,6 +268,11 @@ The default configuration of TinkerGraph remains non-transactional. NOTE: This feature was first made available in TinkerPop 3.7.0 as `TinkerTransactionGraph`. The class was renamed to `TinkerStorageGraph` in 4.0.0. +`TinkerStorageGraph` requires no storage configuration to provide transactions. With no storage engine set it creates +no files and writes nothing to disk, which suits testing and any other case that calls for transaction semantics +without durable state. Configuring a storage engine adds durability to that same graph and is described in the +<>. + ==== Transaction Semantics `TinkerStorageGraph` only has support for `ThreadLocal` transactions, so embedded graph transactions may not be fully @@ -424,6 +419,107 @@ g.V().valueMap() <4> Add a second vertex without committing <5> Rollback the change +[[tinkergraph-gremlin-persistence]] +=== Persistence + +`TinkerStorageGraph` can durably persist to disk through a pluggable storage layer built on its transaction support. +When a storage engine is configured, the changeset of each committed transaction is written to disk, and the graph is +rebuilt from that data when it is opened again. The in-memory `TinkerMemoryGraph` does not retain data across restarts. + +A storage engine is selected with the `gremlin.tinkergraph.storage` configuration key, and +`gremlin.tinkergraph.storage.directory` names the directory that holds the durable data. The value of the storage key is +either a built-in engine name or the fully qualified class name of a `TinkerStorage` implementation, following the same +enum-name-or-class-name convention as the `IdManager` settings. The built-in `graphbinary` engine records committed +transactions as an append-only log serialized with GraphBinary and folds that log into a compact snapshot when the +graph is closed. + +[source,groovy] +---- +conf = new BaseConfiguration() +conf.setProperty("gremlin.graph", "org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph") +conf.setProperty("gremlin.tinkergraph.storage", "graphbinary") +conf.setProperty("gremlin.tinkergraph.storage.directory", "/data/mygraph") + +graph = TinkerStorageGraph.open(conf) +g = traversal().with(graph) +g.addV("person").property("name","marko").iterate() +g.tx().commit() +graph.close() + +// reopening the same location restores the committed data +graph = TinkerStorageGraph.open(conf) +g = traversal().with(graph) +g.V().count() +==>1 +---- + +The graph remains the authoritative in-memory copy and is mirrored to disk, so a persisted graph must still fit in +memory. The write to the storage log happens before the in-memory commit is applied, so a failure to persist aborts the +transaction and leaves the on-disk data and the in-memory graph consistent. + +The durability of each commit is governed by the `gremlin.tinkergraph.storage.sync` setting. Under the default +`commit` mode every committed transaction is forced to disk, so an acknowledged commit survives an operating system +crash or power loss. The `os` mode instead flushes only to the operating system, which is faster but leaves a commit +recoverable only across a crash of the JVM process, not an operating system crash or power loss. + +A storage engine that appends each commit to a log reclaims space by compacting that log into a fresh snapshot of the +committed state. Compaction runs when the graph is closed and can be requested explicitly through +`TinkerStorageGraph.compact()`. So that a long-running graph which is never explicitly closed does not accumulate an +unbounded log, compaction also runs automatically once the log grows past `gremlin.tinkergraph.storage.compactThreshold` +bytes. Setting that threshold to `0` disables automatic compaction. + +The property keys a graph indexes are recorded alongside its data and the indexes are rebuilt when the graph is +opened again, so an index created with `createIndex()` survives a restart and covers the data that was already +stored. An index dropped with `dropIndex()` stays dropped. Index definitions are held outside the transaction log +because an index only affects how fast a lookup runs and never its result, so a definition lost to a crash costs a +rebuild rather than any data. If the record of them cannot be read the graph still opens, with no indexes. + +Element and edge ids are always preserved across a reopen. Auto-generated vertex-property ids are not, by default, +so a vertex property may receive a different id after a reopen. Setting `gremlin.tinkergraph.storage.preserveVertexPropertyIds` +to `true` persists those ids as well, at the cost of a larger store. + +Rebuilding the graph on reopen is subject to `gremlin.tinkergraph.defaultVertexPropertyCardinality`, exactly as +importing data through the `io()` step is (see <>). TinkerGraph has a single +graph-wide cardinality rather than a per-key or per-property one. A graph that holds `list` or `set` multi-properties +must therefore set this default to `list` or `set`. Left at the default `single`, only the last value of each property +survives the reopen. A single stored value cannot be assumed to be a `list` or `set` of one, and the cardinality a +property was written with is not recorded, so the graph-wide default is the only cardinality applied on reopen. This is +the same rule that already governs interchange formats. + +A storage location may be opened by only one graph at a time. `TinkerStorageGraph` takes an exclusive lock on the +storage directory when it opens, so a second attempt to open the same location, whether from the same JVM or another +process, fails rather than corrupting the data. The lock is released when the graph is closed. A store also records +the storage format version it was written with. Opening a store written in a format this version cannot read fails +with a clear error rather than misreading the data. There is no in-place format migration. To move a graph across an +incompatible storage format, export it with the `io()` step before upgrading and read it back afterward. + +Gremlin Server persists a graph the same way. A graph in the server's `graphs` configuration is pointed at a +properties file that sets the storage keys, and the server opens that graph at startup, commits to it as clients +request, and closes it on shutdown. A graph configured without a storage engine remains transactional but in-memory, +so its data is lost when the server stops. + +[source,properties] +---- +gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph +gremlin.tinkergraph.storage=graphbinary +gremlin.tinkergraph.storage.directory=/data/mygraph +---- + +[source,yaml] +---- +graphs: { + graph: conf/tinkerstoragegraph-persistent.properties } +---- + +Persistence is distinct from interchange. `TinkerMemoryGraph` is purely in-memory and does not persist. To move data in +or out of any TinkerGraph in an interchange format such as GraphML, GraphSON, or Gryo, use the `io()` step directly: + +[source,groovy] +---- +g.io("/tmp/graph.kryo").write().iterate() +g.io("/tmp/graph.kryo").read().iterate() +---- + [[tinkergraph-gql]] === TinkerGQL diff --git a/docs/src/upgrade/release-4.x.x.asciidoc b/docs/src/upgrade/release-4.x.x.asciidoc index 36a01282cca..b9453716bf7 100644 --- a/docs/src/upgrade/release-4.x.x.asciidoc +++ b/docs/src/upgrade/release-4.x.x.asciidoc @@ -71,6 +71,67 @@ which affects providers that extended them. See: link:https://lists.apache.org/thread/2zt62kvfssh6xz5vnf2lk1g7cstq9vod[DISCUSS thread] +==== TinkerGraph Disk Storage + +TinkerGraph has always been a pure in-memory graph with some limited capability to write to disk via the `io()` step +or through that same function on close. This approach kept TinkerGraph simple, but limited its functionality for certain +use cases. For TinkerPop 4, TinkerGraph offers a basic file-based persistence option tied to its transaction capability. +This new feature not only gives TinkerGraph another operational dimension for production use cases, but also makes its +transactional capabilities have more purpose. + +The feature is offered via `TinkerStorageGraph` which gained the optional disk storage hinted by its rename. A storage +engine is selected with the new `gremlin.tinkergraph.storage` configuration key, and +`gremlin.tinkergraph.storage.directory` names the directory that holds the durable data. When a storage engine is +configured, each committed transaction is durably written to disk and the graph is rebuilt from that data when it is +opened again, so a graph survives a restart of the JVM. + +The reference engine, `graphbinary`, records committed transactions as an append-only log serialized with GraphBinary +and folds that log into a compact snapshot on close. The storage layer is pluggable: the value of the storage key may +also be the fully-qualified class name of a custom engine, following the same convention as the `IdManager` selection. + +[source,groovy] +---- +conf = new BaseConfiguration() +conf.setProperty('gremlin.graph', 'org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph') +conf.setProperty('gremlin.tinkergraph.storage', 'graphbinary') +conf.setProperty('gremlin.tinkergraph.storage.directory', '/data/mygraph') + +graph = TinkerStorageGraph.open(conf) +g = traversal().with(graph) +g.addV('person').property('name','marko').iterate() +g.tx().commit() +graph.close() + +// reopening the same location restores the committed data +graph = TinkerStorageGraph.open(conf) +g = traversal().with(graph) +c = g.V().count().next() +---- + +The in-memory `TinkerMemoryGraph` no longer persists to disk. Earlier versions of TinkerGraph would automatically read +from `gremlin.tinkergraph.graphLocation` on open and write back to it on close, using the `gremlin.tinkergraph.graphFormat` +interchange format. That automatic behavior is removed, and `TinkerMemoryGraph` now ignores both keys and reports +`FEATURE_PERSISTENCE` as `false`. Note that `gremlin.tinkergraph.graphLocation` named a *file* to be read and written +in an interchange format, while the new `gremlin.tinkergraph.storage.directory` names a *directory* managed by a +storage engine. They are deliberately different keys because they mean different things. Durable persistence is the +responsibility of `TinkerStorageGraph` and its storage engine, while moving data in and out of any graph in an +interchange format remains the job of the `io()` step: + +[source,groovy] +---- +// interchange, on demand, for any graph +g.io('/tmp/graph.kryo').write().iterate() +g.io('/tmp/graph.kryo').read().iterate() +---- + +Configurations that previously relied on the in-memory graph loading itself from `graphLocation` on open must either +call `io().read()` explicitly or switch to `TinkerStorageGraph` with a storage engine. Both +`gremlin.tinkergraph.graphLocation` and `gremlin.tinkergraph.graphFormat` are retired as TinkerGraph settings; no graph +implementation reads either one. They survive only in Gremlin Server's `SimpleAuthenticator`, which uses them to read +its credential store at startup. + +See: <> + === Upgrading for Providers ==== Graph System Providers diff --git a/gremlin-console/conf/tinkergraph-gryo.properties b/gremlin-console/conf/tinkergraph-storage.properties similarity index 60% rename from gremlin-console/conf/tinkergraph-gryo.properties rename to gremlin-console/conf/tinkergraph-storage.properties index 4c2684237c6..9ade10d3d93 100644 --- a/gremlin-console/conf/tinkergraph-gryo.properties +++ b/gremlin-console/conf/tinkergraph-storage.properties @@ -15,7 +15,12 @@ # specific language governing permissions and limitations # under the License. -gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph +# Sample configuration for a durable, transactional TinkerStorageGraph. Opening a graph with this configuration +# (for example via GraphFactory) produces a TinkerStorageGraph that persists committed transactions to the +# configured storage directory. +gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph -gremlin.tinkergraph.graphFormat=gryo -gremlin.tinkergraph.graphLocation=/tmp/tinkergraph.kryo +# built-in storage engine; without this key the graph is transactional but in-memory only +gremlin.tinkergraph.storage=graphbinary +# directory holding the durable data (created if absent; a location may be opened by only one graph at a time) +gremlin.tinkergraph.storage.directory=/tmp/tinkergraph diff --git a/gremlin-server/conf/tinkergraph-credentials.properties b/gremlin-server/conf/tinkergraph-credentials.properties index 4597fabcd3e..a39ff4158f1 100644 --- a/gremlin-server/conf/tinkergraph-credentials.properties +++ b/gremlin-server/conf/tinkergraph-credentials.properties @@ -16,5 +16,9 @@ # under the License. gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph gremlin.tinkergraph.vertexIdManager=LONG +# This credential store is an in-memory TinkerGraph. TinkerGraph no longer auto-loads from disk on open, so +# SimpleAuthenticator reads the store explicitly from graphLocation using graphFormat at startup. Both keys are +# retired as TinkerGraph settings and are honoured only for this load; they name an interchange FILE, not the +# storage DIRECTORY that a durable TinkerStorageGraph configures with gremlin.tinkergraph.storage.directory. gremlin.tinkergraph.graphLocation=data/credentials.kryo gremlin.tinkergraph.graphFormat=gryo \ No newline at end of file diff --git a/gremlin-server/conf/tinkerstoragegraph-persistent.properties b/gremlin-server/conf/tinkerstoragegraph-persistent.properties new file mode 100644 index 00000000000..4d1ad1de7cb --- /dev/null +++ b/gremlin-server/conf/tinkerstoragegraph-persistent.properties @@ -0,0 +1,40 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Sample configuration for a durable, transactional TinkerStorageGraph on Gremlin Server. Reference it from the +# server's "graphs" block, for example in gremlin-server-transaction.yaml: +# graphs: { graph: conf/tinkerstoragegraph-persistent.properties } +gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph + +# Enables durable persistence. "graphbinary" is the built-in storage engine; the fully qualified class name of a +# custom TinkerStorage implementation may be used instead. Without this key the graph is transactional but in-memory. +gremlin.tinkergraph.storage=graphbinary +# Directory holding the durable data. Required when a storage engine is set. It is created if absent and may be +# opened by only one graph at a time (single writer, whether in this JVM or another process). +gremlin.tinkergraph.storage.directory=/tmp/tinkerstoragegraph + +gremlin.tinkergraph.vertexIdManager=LONG +gremlin.tinkergraph.edgeIdManager=LONG +gremlin.tinkergraph.vertexPropertyIdManager=LONG + +# Optional storage tuning (defaults shown, uncomment to change): +# "commit" forces each transaction to disk (survives OS crash/power loss); "os" is faster but survives only a JVM crash +#gremlin.tinkergraph.storage.sync=commit +# auto-compact the append log once it grows past this many bytes (default 67108864 = 64MB; 0 disables) +#gremlin.tinkergraph.storage.compactThreshold=67108864 +# persist auto-generated vertex-property ids across a reopen (default false) +#gremlin.tinkergraph.storage.preserveVertexPropertyIds=false diff --git a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/auth/SimpleAuthenticator.java b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/auth/SimpleAuthenticator.java index afdae0d2de7..28c82e93983 100644 --- a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/auth/SimpleAuthenticator.java +++ b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/auth/SimpleAuthenticator.java @@ -21,14 +21,17 @@ import org.apache.tinkerpop.gremlin.groovy.jsr223.dsl.credential.CredentialTraversal; import org.apache.tinkerpop.gremlin.groovy.jsr223.dsl.credential.CredentialTraversalDsl; import org.apache.tinkerpop.gremlin.groovy.jsr223.dsl.credential.CredentialTraversalSource; +import org.apache.commons.configuration2.Configuration; import org.apache.tinkerpop.gremlin.structure.Graph; import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.structure.io.IoCore; import org.apache.tinkerpop.gremlin.structure.util.GraphFactory; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; import org.mindrot.jbcrypt.BCrypt; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.io.File; import java.net.InetAddress; import java.nio.charset.StandardCharsets; import java.util.Arrays; @@ -80,12 +83,63 @@ public void setup(final Map config) { // have to create the indices because they are not stored in gryo final TinkerGraph tinkerGraph = (TinkerGraph) graph; tinkerGraph.createIndex(PROPERTY_USERNAME, Vertex.class); + + // TinkerGraph no longer auto-loads from graphLocation on open, so read the credential store here from + // the configured location/format. TinkerStorageGraph (which persists via its own storage engine) manages + // its own data and is left untouched. + loadCredentialStore(tinkerGraph); } credentialStore = graph.traversal(CredentialTraversalSource.class); logger.info("CredentialGraph initialized at {}", credentialStore); } + /** + * Reads the credential store into the supplied in-memory {@link TinkerGraph} from the + * {@code gremlin.tinkergraph.graphLocation} and {@code gremlin.tinkergraph.graphFormat} entries of its + * configuration, if present. Earlier versions of TinkerGraph read those keys themselves on open, so the + * credential store loaded as a side effect of {@code GraphFactory.open}; that automatic behaviour was removed + * and this method preserves it for the credential store alone. + *

+ * No TinkerGraph reads either key any more, which is why they are named here as literals rather than through + * constants. Despite the {@code gremlin.tinkergraph} prefix they are in effect settings of this authenticator, + * and belong in its own {@code config} block beside {@code credentialsDb} rather than in the graph's properties + * file. They are left in place here only to keep existing credential configurations working. A graph with a + * storage engine manages its own data and is skipped. + */ + private static void loadCredentialStore(final TinkerGraph graph) { + final Configuration conf = graph.configuration(); + final String location = conf.getString("gremlin.tinkergraph.graphLocation", null); + // a storage engine manages its own persistence and is not an interchange-format load + final String storage = conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, null); + if (null == location || storage != null) + return; + + final File f = new File(location); + if (!f.exists() || !f.isFile()) + return; + + final String format = conf.getString("gremlin.tinkergraph.graphFormat", "gryo"); + try { + switch (format) { + case "graphml": + graph.io(IoCore.graphml()).readGraph(location); + break; + case "graphson": + graph.io(IoCore.graphson()).readGraph(location); + break; + case "gryo": + graph.io(IoCore.gryo()).readGraph(location); + break; + default: + graph.io(IoCore.createIoBuilder(format)).readGraph(location); + break; + } + } catch (Exception ex) { + throw new IllegalStateException(String.format("Could not load credential store at %s with format %s", location, format), ex); + } + } + @Override public SaslNegotiator newSaslNegotiator(final InetAddress remoteAddress) { return new PlainTextSaslAuthenticator(); diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/AbstractTinkerGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/AbstractTinkerGraph.java index fac8d87b2df..336eaa83ae4 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/AbstractTinkerGraph.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/AbstractTinkerGraph.java @@ -28,7 +28,6 @@ import org.apache.tinkerpop.gremlin.structure.Vertex; import org.apache.tinkerpop.gremlin.structure.VertexProperty; import org.apache.tinkerpop.gremlin.structure.io.Io; -import org.apache.tinkerpop.gremlin.structure.io.IoCore; import org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONVersion; import org.apache.tinkerpop.gremlin.structure.io.gryo.GryoVersion; import org.apache.tinkerpop.gremlin.structure.util.StringFactory; @@ -36,8 +35,10 @@ import org.apache.tinkerpop.gremlin.tinkergraph.process.computer.TinkerGraphComputerView; import org.apache.tinkerpop.gremlin.gql.GqlDeclarativeMatchStrategy; import org.apache.tinkerpop.gremlin.tinkergraph.services.TinkerServiceRegistry; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.DefaultStorage; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.DirectoryLock; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorage; -import java.io.File; import java.lang.reflect.InvocationTargetException; import java.util.Collections; import java.util.Iterator; @@ -45,6 +46,7 @@ import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.atomic.AtomicLong; +import java.util.concurrent.locks.ReentrantLock; /** * Base class for {@link TinkerMemoryGraph} and {@link TinkerStorageGraph}. @@ -76,8 +78,39 @@ public abstract class AbstractTinkerGraph implements TinkerGraph { protected TinkerServiceRegistry serviceRegistry; protected Configuration configuration; - protected String graphLocation; - protected String graphFormat; + + /** + * The filesystem directory backing the storage engine, from {@code gremlin.tinkergraph.storage.directory}, or + * {@code null} when the graph holds data only in memory. + */ + protected String storageDirectory; + + /** + * The pluggable durable storage engine, or {@code null} when the graph holds data only in memory. Only set by + * transactional implementations that support persistence. + */ + protected TinkerStorage storage; + + /** + * Exclusive lock on the storage directory, held for the graph's lifetime so no second graph — in this or another + * process — can open the same location and corrupt its files. {@code null} when the graph is purely in-memory. + */ + protected DirectoryLock directoryLock; + + /** + * Serializes the durable write of a committing transaction. TinkerGraph transactions lock only their own changed + * elements, so two commits touching disjoint elements run their commit paths concurrently; without this lock they + * would both write to the storage engine's single append log at once and interleave (corrupt) its records. Held + * only around the engine's persist/flush, so commits of disjoint elements still proceed in parallel up to that + * point. Fair, so committers are served in arrival order and none is starved. + */ + protected final ReentrantLock storageCommitLock = new ReentrantLock(true); + + /** + * Guard set while a graph is replaying its storage log on open. While {@code true}, mutations must not be + * re-persisted, otherwise replay would append the loaded data back to the log. + */ + protected volatile boolean loading = false; /** * {@inheritDoc} @@ -243,54 +276,6 @@ public boolean hasVertexProperty(final Object id) { return vertexProperties.containsKey(id); } - protected void loadGraph() { - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) { - try { - if (graphFormat.equals("graphml")) { - io(IoCore.graphml()).readGraph(graphLocation); - } else if (graphFormat.equals("graphson")) { - io(IoCore.graphson()).readGraph(graphLocation); - } else if (graphFormat.equals("gryo")) { - io(IoCore.gryo()).readGraph(graphLocation); - } else { - io(IoCore.createIoBuilder(graphFormat)).readGraph(graphLocation); - } - } catch (Exception ex) { - throw new RuntimeException(String.format("Could not load graph at %s with %s", graphLocation, graphFormat), ex); - } - } - } - - protected void saveGraph() { - final File f = new File(graphLocation); - if (f.exists()) { - f.delete(); - } else { - final File parent = f.getParentFile(); - - // the parent would be null in the case of an relative path if the graphLocation was simply: "f.gryo" - if (parent != null && !parent.exists()) { - parent.mkdirs(); - } - } - - try { - if (graphFormat.equals("graphml")) { - io(IoCore.graphml()).writeGraph(graphLocation); - } else if (graphFormat.equals("graphson")) { - io(IoCore.graphson()).writeGraph(graphLocation); - } else if (graphFormat.equals("gryo")) { - io(IoCore.gryo()).writeGraph(graphLocation); - } else { - io(IoCore.createIoBuilder(graphFormat)).writeGraph(graphLocation); - } - } catch (Exception ex) { - throw new RuntimeException(String.format("Could not save graph at %s with %s", graphLocation, graphFormat), ex); - } - } - - @Override public I io(final Io.Builder builder) { if (builder.requiresVersion(GryoVersion.V1_0) || builder.requiresVersion(GraphSONVersion.V1_0)) @@ -337,13 +322,31 @@ public void clear() { } /** - * This method only has an effect if the {@link TinkerGraph#GREMLIN_TINKERGRAPH_GRAPH_LOCATION} is set, in which case the - * data in the graph is persisted to that location. This method may be called multiple times and does not release - * resources. + * Closes the graph, releasing any resources held by its {@link TinkerServiceRegistry}. This method may be called + * multiple times and is a no-op with respect to graph data for the in-memory implementation. Transactional + * implementations that are backed by a {@link org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorage} + * engine flush and close that engine here. */ @Override public void close() { - if (graphLocation != null) saveGraph(); + if (storage != null) { + // serialize against concurrent commit writes: close flushes, compacts, and closes the log, which must not + // interleave with a transaction appending to it. + storageCommitLock.lock(); + try { + storage.flush(); + storage.compact(this); + storage.close(); + } finally { + storageCommitLock.unlock(); + // release the exclusive directory lock last, so the location is only reopenable once the engine has + // fully released its files + if (directoryLock != null) { + directoryLock.close(); + directoryLock = null; + } + } + } serviceRegistry.close(); GqlDeclarativeMatchStrategy.evict(this); } @@ -555,6 +558,46 @@ protected static IdManager selectIdManager(final Configur } } + ///////////// Storage engine /////////////// + /** + * The committed vertices of the graph, for a storage engine to snapshot during compaction. Unlike {@link #vertices()}, + * this view excludes any uncommitted transaction-local state, so compaction never persists changes that a caller has + * not committed. The base implementation, which has no transactional isolation, is equivalent to {@link #vertices()}; + * a transactional subclass overrides it to read only committed element state. + */ + public Iterator committedVertices() { + return vertices(); + } + + /** + * The committed edges of the graph, for a storage engine to snapshot during compaction. The edge counterpart of + * {@link #committedVertices()}. + */ + public Iterator committedEdges() { + return edges(); + } + + /** + * Construct a {@link TinkerStorage} engine from the TinkerGraph {@code Configuration}, or return {@code null} when + * no storage engine is configured. The configuration value is either a {@link DefaultStorage} enum name (matched + * case-insensitively, e.g. {@code graphbinary}) or the fully-qualified class name of a {@link TinkerStorage} + * implementation with a public no-argument constructor. Mirrors {@link #selectIdManager}. + */ + protected static TinkerStorage selectStorage(final Configuration config, final String configKey) { + final String storageConfigValue = config.getString(configKey, null); + if (null == storageConfigValue) + return null; + try { + return DefaultStorage.valueOf(storageConfigValue.toUpperCase()).get(); + } catch (IllegalArgumentException iae) { + try { + return (TinkerStorage) Class.forName(storageConfigValue).newInstance(); + } catch (Exception ex) { + throw new IllegalStateException(String.format("Could not configure TinkerGraph storage engine with %s", storageConfigValue), ex); + } + } + } + protected TinkerServiceRegistry.TinkerServiceFactory instantiate(final String className) { try { return (TinkerServiceRegistry.TinkerServiceFactory) Class.forName(className).getConstructor(AbstractTinkerGraph.class).newInstance(this); diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java index dc931b9061a..0f7c0088b72 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java @@ -45,8 +45,44 @@ public interface TinkerGraph extends Graph { String GREMLIN_TINKERGRAPH_EDGE_ID_MANAGER = "gremlin.tinkergraph.edgeIdManager"; String GREMLIN_TINKERGRAPH_VERTEX_PROPERTY_ID_MANAGER = "gremlin.tinkergraph.vertexPropertyIdManager"; String GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY = "gremlin.tinkergraph.defaultVertexPropertyCardinality"; - String GREMLIN_TINKERGRAPH_GRAPH_LOCATION = "gremlin.tinkergraph.graphLocation"; - String GREMLIN_TINKERGRAPH_GRAPH_FORMAT = "gremlin.tinkergraph.graphFormat"; + /** + * Selects the pluggable storage engine used by {@link TinkerStorageGraph} to durably persist transactions to the + * {@link #GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY} directory. The value is either a + * {@code TinkerStorageGraph.DefaultStorage} enum name (e.g. {@code graphbinary}) or the fully-qualified class name + * of a {@code org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorage} implementation. When unset, + * the graph holds data only in memory. Not valid on {@link TinkerMemoryGraph}. + */ + String GREMLIN_TINKERGRAPH_STORAGE = "gremlin.tinkergraph.storage"; + /** + * The filesystem directory that a {@link TinkerStorageGraph} storage engine uses for its durable data. Ignored by + * {@link TinkerMemoryGraph}, which is purely in-memory. Only meaningful when {@link #GREMLIN_TINKERGRAPH_STORAGE} + * is also set. + */ + String GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY = "gremlin.tinkergraph.storage.directory"; + /** + * The durability mode a {@link TinkerStorageGraph} storage engine applies on commit. Either {@code commit} + * (default) to {@code fsync} every commit so acknowledged commits survive an OS crash or power loss, or {@code os} + * to only flush to the operating system so commits survive a JVM process crash but may be lost on OS crash or + * power loss. Only meaningful when {@link #GREMLIN_TINKERGRAPH_STORAGE} is set. See + * {@code org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.SyncMode}. + */ + String GREMLIN_TINKERGRAPH_STORAGE_SYNC = "gremlin.tinkergraph.storage.sync"; + /** + * The size in bytes at which a {@link TinkerStorageGraph} storage engine automatically compacts its append log on + * commit, bounding the log growth (and restart replay cost) of a long-running graph that is never explicitly + * closed. Defaults to 67108864 (64 MB). Set to {@code 0} to disable automatic compaction and rely on + * {@code close()} or an explicit {@code compact()}. Only meaningful when {@link #GREMLIN_TINKERGRAPH_STORAGE} is + * set. + */ + String GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD = "gremlin.tinkergraph.storage.compactThreshold"; + /** + * Whether a {@link TinkerStorageGraph} storage engine persists auto-generated vertex-property ids so they are + * stable across a close and reopen. Defaults to {@code false}: vertex-property ids are regenerated on load, which + * keeps the store smaller. Element and edge ids are always preserved regardless of this setting. Each record is + * self-describing, so a store written with this enabled reopens correctly even if the setting later differs. Only + * meaningful when {@link #GREMLIN_TINKERGRAPH_STORAGE} is set. + */ + String GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS = "gremlin.tinkergraph.storage.preserveVertexPropertyIds"; String GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES = "gremlin.tinkergraph.allowNullPropertyValues"; String GREMLIN_TINKERGRAPH_SERVICE = "gremlin.tinkergraph.service"; String GREMLIN_TINKERGRAPH_VERTEX_LABEL_CARDINALITY = "gremlin.tinkergraph.vertexLabelCardinality"; diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraph.java index fc1afcad33e..8dd7b1671f3 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraph.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraph.java @@ -50,8 +50,10 @@ import java.util.concurrent.atomic.AtomicInteger; /** - * The in-memory (with optional persistence on calls to {@link #close()}) implementation of the {@link TinkerGraph} - * interface and the reference implementation of the property graph interfaces provided by TinkerPop. + * The purely in-memory implementation of the {@link TinkerGraph} interface and the reference implementation of the + * property graph interfaces provided by TinkerPop. This graph holds no data across JVM restarts; use + * {@code g.io(...).write()} / {@code g.io(...).read()} for interchange, or {@link TinkerStorageGraph} for durable + * persistence. * * @author Marko A. Rodriguez (http://markorodriguez.com) * @author Stephen Mallette (http://stephen.genoprime.com) @@ -99,15 +101,6 @@ public class TinkerMemoryGraph extends AbstractTinkerGraph { defaultVertexLabel = Vertex.DEFAULT_LABEL; defaultEdgeLabel = Edge.DEFAULT_LABEL; - graphLocation = configuration.getString(GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null); - graphFormat = configuration.getString(GREMLIN_TINKERGRAPH_GRAPH_FORMAT, null); - - if ((graphLocation != null && null == graphFormat) || (null == graphLocation && graphFormat != null)) - throw new IllegalStateException(String.format("The %s and %s must both be specified if either is present", - GREMLIN_TINKERGRAPH_GRAPH_LOCATION, GREMLIN_TINKERGRAPH_GRAPH_FORMAT)); - - if (graphLocation != null) loadGraph(); - serviceRegistry = new TinkerServiceRegistry(this); configuration.getList(String.class, GREMLIN_TINKERGRAPH_SERVICE, Collections.emptyList()).forEach(serviceClass -> serviceRegistry.registerService(instantiate(serviceClass))); @@ -447,6 +440,11 @@ public boolean supportsThreadedTransactions() { return false; } + @Override + public boolean supportsPersistence() { + return false; + } + @Override public boolean supportsServiceCall() { return true; diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java index c74c863e349..594c242c3e3 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java @@ -35,8 +35,11 @@ import org.apache.tinkerpop.gremlin.tinkergraph.process.traversal.strategy.optimization.TinkerGraphCountStrategy; import org.apache.tinkerpop.gremlin.tinkergraph.process.traversal.strategy.optimization.TinkerGraphStepStrategy; import org.apache.tinkerpop.gremlin.tinkergraph.services.TinkerServiceRegistry; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.DirectoryLock; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.IndexDefinitions; import org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils; +import java.io.File; import java.util.Arrays; import java.util.Collections; import java.util.Iterator; @@ -47,9 +50,17 @@ import java.util.concurrent.ConcurrentHashMap; /** - * The transactional implementation of the {@link TinkerGraph} interface, in-memory with optional persistence on - * calls to {@link #close()}. It is planned that this implementation will optionally support simple storage to disk - * built on its transaction functionality. + * The transactional implementation of the {@link TinkerGraph} interface. It provides {@code read committed} + * transaction isolation with optimistic locking and, when a storage engine is configured, durable persistence to + * disk. With no storage engine configured it is an in-memory transactional graph that retains nothing across + * restarts. + *

+ * Persistence is pluggable through the {@link org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorage} + * SPI and enabled with the {@code gremlin.tinkergraph.storage} and {@code gremlin.tinkergraph.storage.directory} + * configuration keys. Each committed transaction is written through to the storage engine before the in-memory commit + * is applied, and reopening the same location replays the persisted commits to rebuild the graph. A storage location + * is single-writer: it is guarded by an exclusive {@link org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.DirectoryLock} + * so a second open of the same directory fails rather than corrupting the data. * * @author Valentyn Kahamlyk */ @@ -74,6 +85,12 @@ public final class TinkerStorageGraph extends AbstractTinkerGraph { private final TinkerTransaction transaction = new TinkerTransaction(this); + /** + * Set while indexes recorded for the store are being recreated on open, so applying them does not rewrite the + * file they were just read from. + */ + private boolean restoringIndexes = false; + private final Map> vertices = new ConcurrentHashMap<>(); private final Map> edges = new ConcurrentHashMap<>(); @@ -95,18 +112,42 @@ private TinkerStorageGraph(final Configuration configuration) { defaultVertexLabel = Vertex.DEFAULT_LABEL; defaultEdgeLabel = Edge.DEFAULT_LABEL; - graphLocation = configuration.getString(GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null); - graphFormat = configuration.getString(GREMLIN_TINKERGRAPH_GRAPH_FORMAT, null); + storageDirectory = configuration.getString(GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, null); + storage = selectStorage(configuration, GREMLIN_TINKERGRAPH_STORAGE); - if ((graphLocation != null && null == graphFormat) || (null == graphLocation && graphFormat != null)) - throw new IllegalStateException(String.format("The %s and %s must both be specified if either is present", - GREMLIN_TINKERGRAPH_GRAPH_LOCATION, GREMLIN_TINKERGRAPH_GRAPH_FORMAT)); - - if (graphLocation != null) loadGraph(); + if (storage != null && null == storageDirectory) + throw new IllegalStateException(String.format("The %s must be specified when %s is set", + GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, GREMLIN_TINKERGRAPH_STORAGE)); serviceRegistry = new TinkerServiceRegistry(this); configuration.getList(String.class, GREMLIN_TINKERGRAPH_SERVICE, Collections.emptyList()).forEach(serviceClass -> serviceRegistry.registerService(instantiate(serviceClass))); + + if (storage != null) { + // take an exclusive lock on the storage directory before the engine touches any files, so a second graph + // on the same location fails fast rather than corrupting it. The directory must exist to hold the lock. + final File dir = new File(storageDirectory); + if (!dir.isDirectory() && !dir.mkdirs()) + throw new IllegalStateException(String.format("Could not create storage directory %s", dir)); + directoryLock = DirectoryLock.acquire(dir); + try { + storage.open(this, configuration); + loading = true; + try { + storage.replay(this); + } finally { + loading = false; + } + // recreate the recorded indexes now that replay has rebuilt the elements they cover, so + // createKeyIndex backfills over the restored data rather than only over writes that follow + restoreIndexes(dir); + } catch (RuntimeException | Error ex) { + // don't leak the lock if the engine fails to open or replay + directoryLock.close(); + directoryLock = null; + throw ex; + } + } } /** @@ -289,6 +330,24 @@ public void clear() { this.edges.clear(); } + /** + * Fold the durable storage log into a compact snapshot of the current committed state, reclaiming space. Has no + * effect when no storage engine is configured. + */ + public void compact() { + if (storage != null) { + // hold the same lock as the commit write path: compaction closes the log, rewrites the snapshot, and + // truncates the log, which must not interleave with a concurrent transaction appending to that log. + storageCommitLock.lock(); + try { + storage.flush(); + storage.compact(this); + } finally { + storageCommitLock.unlock(); + } + } + } + @Override public Transaction tx() { return transaction; @@ -311,6 +370,20 @@ public boolean hasVertex(Object id) { Map> getVertices () { return vertices; } + /** + * {@inheritDoc} + *

+ * Reads the committed value of each container ({@link TinkerElementContainer#getUnmodified()}), so a compaction + * snapshot reflects only committed state regardless of the calling thread's open transaction. A container whose + * committed value is {@code null} (added but not yet committed, or committed and then deleted) is skipped. + */ + @Override + public Iterator committedVertices() { + return IteratorUtils.map( + IteratorUtils.filter(vertices.values().iterator(), c -> c.getUnmodified() != null), + c -> (Vertex) c.getUnmodified()); + } + @Override public int getEdgesCount() { return (int) edges.entrySet().stream().filter(v -> v.getValue().get() != null).count(); @@ -339,6 +412,19 @@ public boolean hasEdge(Object id) { Map> getEdges () { return edges; } + /** + * {@inheritDoc} + *

+ * The edge counterpart of {@link #committedVertices()}: reads only committed container state and skips containers + * whose committed value is {@code null}. + */ + @Override + public Iterator committedEdges() { + return IteratorUtils.map( + IteratorUtils.filter(edges.values().iterator(), c -> c.getUnmodified() != null), + c -> (Edge) c.getUnmodified()); + } + @Override public TinkerServiceRegistry getServiceRegistry() { return serviceRegistry; @@ -504,6 +590,12 @@ public class TinkerGraphGraphFeatures implements Features.GraphFeatures { private TinkerGraphGraphFeatures() { } + /** + * A persistent {@link TinkerStorageGraph} is a single-writer store: {@code DirectoryLock} permits only one + * graph instance to open a given storage directory at a time. This feature denotes multiple connections / + * instances sharing the same data — not the intra-instance, multi-thread transaction access that the + * thread-local {@link TinkerTransaction} already provides — so it is {@code false}. + */ @Override public boolean supportsConcurrentAccess() { return false; @@ -519,6 +611,11 @@ public boolean supportsTransactions() { return true; } + @Override + public boolean supportsPersistence() { + return storage != null; + } + @Override public boolean supportsServiceCall() { return true; @@ -549,6 +646,7 @@ public void createIndex(final String key, final Class ele } else { throw new IllegalArgumentException("Class is not indexable: " + elementClass); } + recordIndexes(); } /** @@ -567,5 +665,36 @@ public void dropIndex(final String key, final Class eleme } else { throw new IllegalArgumentException("Class is not indexable: " + elementClass); } + recordIndexes(); + } + + /** + * Recreate the indexes recorded for this store. Runs after replay so that {@code createKeyIndex} backfills over + * the elements it has just rebuilt. The definitions are already on disk, so recording is suppressed while they + * are applied. + */ + private void restoreIndexes(final File directory) { + final IndexDefinitions definitions = IndexDefinitions.read(directory); + if (definitions.isEmpty()) + return; + restoringIndexes = true; + try { + definitions.vertexKeys().forEach(key -> createIndex(key, Vertex.class)); + definitions.edgeKeys().forEach(key -> createIndex(key, Edge.class)); + } finally { + restoringIndexes = false; + } + } + + /** + * Record the current set of indexed keys beside the engine's files, so a reopen restores them. Index definitions + * are not part of the transactional log; see {@link IndexDefinitions} for why that is sound. A graph with no + * storage engine keeps everything in memory and writes nothing. + */ + private void recordIndexes() { + if (null == storage || restoringIndexes) + return; + new IndexDefinitions(getIndexedKeys(Vertex.class), getIndexedKeys(Edge.class)) + .write(new File(storageDirectory)); } } diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerTransaction.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerTransaction.java index 0bfcf09e469..12f585cff3a 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerTransaction.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerTransaction.java @@ -21,9 +21,12 @@ import org.apache.tinkerpop.gremlin.structure.Transaction; import org.apache.tinkerpop.gremlin.structure.util.AbstractThreadLocalTransaction; import org.apache.tinkerpop.gremlin.structure.util.TransactionException; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorageMutation; +import java.util.ArrayList; import java.util.Collections; import java.util.HashSet; +import java.util.List; import java.util.Set; import java.util.concurrent.atomic.AtomicLong; @@ -174,9 +177,36 @@ protected void doCommit() throws TransactionException { final TinkerTransactionalIndex edgeIndex = (TinkerTransactionalIndex) graph.edgeIndex; if (edgeIndex != null) edgeIndex.commit(changedEdges); - // commit all changes - changedVertices.forEach(v -> v.commit(txVersion)); - changedEdges.forEach(e -> e.commit(txVersion)); + // write-ahead: durably persist the changeset before applying the in-memory commit, so a failure here + // aborts the commit (via the catch below) and leaves memory and disk consistent. Skipped while the graph + // is replaying its storage log on open. Serialized by storageCommitLock because commits of disjoint + // elements otherwise reach the engine's single append log concurrently and interleave its records. + // + // The in-memory apply is held inside that same lock. Compaction builds its snapshot by reading the graph + // and then discards the log, which is only sound while the graph reflects everything the log holds. That + // is false for exactly as long as a changeset sits persisted but not yet applied, so a compaction landing + // in that window snapshots without the transaction and then deletes the record that held it, losing an + // acknowledged commit. Publishing under the lock closes the window, and also makes the order in which + // transactions become visible match the order they were recorded in. + final boolean durable = graph.storage != null && !graph.loading; + if (durable) graph.storageCommitLock.lock(); + try { + if (durable) { + graph.storage.persist(txVersion, toVertexMutations(changedVertices), toEdgeMutations(changedEdges)); + graph.storage.flush(); + } + + // commit all changes + changedVertices.forEach(v -> v.commit(txVersion)); + changedEdges.forEach(e -> e.commit(txVersion)); + + // bound log growth for a long-running graph that is never explicitly closed; no-op unless the + // engine's accumulated log has crossed its threshold. Runs after the apply so the snapshot it may + // write includes this transaction rather than omitting it and then truncating the log that held it. + if (durable) graph.storage.maybeCompact(graph); + } finally { + if (durable) graph.storageCommitLock.unlock(); + } } catch (TransactionException ex) { // rollback on error changedVertices.forEach(v -> v.rollback()); @@ -209,6 +239,34 @@ protected void doCommit() throws TransactionException { } } + /** + * Convert the changed vertex containers into the storage-facing {@link TinkerStorageMutation} view. Called during + * commit, before {@code commit()} is applied to the containers, so the modified value is still available via + * {@link TinkerElementContainer#getModified()}. + */ + private static List> toVertexMutations(final Set> changed) { + final List> mutations = new ArrayList<>(changed.size()); + for (final TinkerElementContainer c : changed) { + mutations.add(c.isDeleted() + ? new TinkerStorageMutation<>(c.getElementId(), null) + : new TinkerStorageMutation<>(c.getElementId(), c.getModified())); + } + return mutations; + } + + /** + * Convert the changed edge containers into the storage-facing {@link TinkerStorageMutation} view. + */ + private static List> toEdgeMutations(final Set> changed) { + final List> mutations = new ArrayList<>(changed.size()); + for (final TinkerElementContainer c : changed) { + mutations.add(c.isDeleted() + ? new TinkerStorageMutation<>(c.getElementId(), null) + : new TinkerStorageMutation<>(c.getElementId(), c.getModified())); + } + return mutations; + } + /** * Rollback all changes made in current transaction. * Workflow: diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractLogStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractLogStorage.java new file mode 100644 index 00000000000..b186471eda8 --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractLogStorage.java @@ -0,0 +1,485 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.util.Attachable; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.AbstractTinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerEdge; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex; + +import java.io.BufferedInputStream; +import java.io.BufferedOutputStream; +import java.io.DataInputStream; +import java.io.DataOutputStream; +import java.io.EOFException; +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.UncheckedIOException; +import java.nio.channels.FileChannel; +import java.nio.file.AtomicMoveNotSupportedException; +import java.nio.file.Files; +import java.nio.file.StandardCopyOption; +import java.nio.file.StandardOpenOption; +import java.util.Arrays; +import java.util.Collection; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.zip.CRC32; + +/** + * Log-structured durable storage machinery shared by {@link TinkerStorage} engines, independent of how an element is + * encoded. It persists a {@code TinkerStorageGraph} as an append-only commit log ({@code log.gbin}) plus an optional + * folded {@code snapshot.gbin}; on open the snapshot is read followed by the log, last-write-wins per element id. + *

+ * This base owns everything that is not the element codec: the on-disk file layout, the single-source-of-truth + * {@code VERSION} marker, length+CRC frame framing (which tells an interrupted trailing append apart from genuine + * corruption), the replay fold loop, durability via {@link SyncMode}, and crash-safe atomic compaction with a + * size threshold. Concrete engines supply only the codec through {@link #encodeCommit}, {@link #decodeFrame}, + * {@link #writeSnapshot}, and (optionally) {@link #beginReplay} for per-replay decode state. + *

+ * The in-memory graph remains authoritative (write-through); this machinery does not support graphs larger than memory. + */ +public abstract class AbstractLogStorage implements TinkerStorage { + + /** + * Magic bytes ("TGSB" — TinkerGraph Storage Binary) at the start of every storage file, so a file can be + * identified as one written by this engine (and a foreign or corrupt file rejected) before any record is read. + */ + static final byte[] MAGIC = { 'T', 'G', 'S', 'B' }; + + /** + * On-disk format version of the store. Recorded once per store in the {@link #VERSION_FILE} marker rather than in + * every file, so a store has a single unambiguous version even when it momentarily holds a snapshot and a log + * written at different times. A future format bump is detected against this marker so an older store is rejected + * (never silently misread); the supported migration path is to export via {@code g.io()} before upgrading. + */ + static final byte FORMAT_VERSION = 1; + + /** + * Store-level version marker file, holding {@link #MAGIC} followed by the one-byte {@link #FORMAT_VERSION}. + */ + static final String VERSION_FILE = "VERSION"; + + /** + * Bytes of the per-file header: just {@link #MAGIC}. The format version lives in the store-level + * {@link #VERSION_FILE}, not in each file. + */ + static final int HEADER_SIZE = MAGIC.length; + + static final String SNAPSHOT_FILE = "snapshot.gbin"; + static final String LOG_FILE = "log.gbin"; + + /** + * Default automatic-compaction threshold: 64 MB of appended log since the last compaction. + */ + static final long DEFAULT_COMPACT_THRESHOLD_BYTES = 64L * 1024 * 1024; + + private File directory; + private File snapshotFile; + private File logFile; + private File versionFile; + + private DataOutputStream logOut; + private FileOutputStream logFos; + private SyncMode syncMode = SyncMode.COMMIT; + private long compactThresholdBytes = DEFAULT_COMPACT_THRESHOLD_BYTES; + private long logBytesSinceCompaction = 0; + private boolean closed = false; + + // ----------------------------------------------------------------------------------------- codec hooks + + /** + * Encode a committing transaction's changeset into a single record payload (the framing is added by the caller). + */ + protected abstract byte[] encodeCommit(long txVersion, + Collection> changedVertices, + Collection> changedEdges) throws IOException; + + /** + * Decode one record payload, folding its puts and deletes into the supplied maps (last-write-wins per id). + */ + protected abstract void decodeFrame(byte[] record, + Map vertices, + Map edges) throws IOException; + + /** + * Write the entire current committed state of the graph to {@code out} as framed records (via {@link #writeFrame}), + * for compaction. The fixed {@link #MAGIC} header has already been written to {@code out}. + */ + protected abstract void writeSnapshot(AbstractTinkerGraph graph, DataOutputStream out) throws IOException; + + /** + * Reset any per-replay decode state (e.g. a dictionary) before a fold begins. Default is a no-op. + */ + protected void beginReplay() { + // no-op by default + } + + // ----------------------------------------------------------------------------------------- lifecycle + + @Override + public void open(final AbstractTinkerGraph graph, final Configuration config) { + final String location = config.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, null); + if (null == location) + throw new IllegalStateException(String.format("%s must be set to use a durable storage engine", + TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY)); + this.directory = new File(location); + this.snapshotFile = new File(directory, SNAPSHOT_FILE); + this.logFile = new File(directory, LOG_FILE); + this.versionFile = new File(directory, VERSION_FILE); + this.syncMode = SyncMode.fromConfigValue(config.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_SYNC, null)); + this.compactThresholdBytes = config.getLong( + TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, DEFAULT_COMPACT_THRESHOLD_BYTES); + // seed the counter with any pre-existing log so a graph reopened with a large log still compacts promptly + this.logBytesSinceCompaction = logFile.exists() ? logFile.length() : 0; + configureCodec(config); + ensureDirectory(); + establishStoreVersion(); + } + + /** + * Read any codec-specific configuration. Called once during {@link #open}. Default is a no-op. + */ + protected void configureCodec(final Configuration config) { + // no-op by default + } + + @Override + public void replay(final AbstractTinkerGraph graph) { + beginReplay(); + // Fold snapshot then log into final state: last write per id wins, deletes remove. + final Map vertices = new LinkedHashMap<>(); + final Map edges = new LinkedHashMap<>(); + + if (snapshotFile.exists()) + foldRecords(snapshotFile, vertices, edges); + if (logFile.exists()) + foldRecords(logFile, vertices, edges); + + if (vertices.isEmpty() && edges.isEmpty()) + return; + + // Attach vertices first so edges can find their endpoints, then commit once. + for (final DetachedVertex v : vertices.values()) + v.attach(Attachable.Method.getOrCreate(graph)); + for (final DetachedEdge e : edges.values()) + e.attach(Attachable.Method.getOrCreate(graph)); + + graph.tx().commit(); + } + + @Override + public void persist(final long txVersion, + final Collection> changedVertices, + final Collection> changedEdges) { + ensureLogOpen(); + try { + final byte[] frame = encodeCommit(txVersion, changedVertices, changedEdges); + writeFrame(logOut, frame); + logBytesSinceCompaction += 2L * Integer.BYTES + frame.length; // length + crc prefixes + payload + } catch (IOException ex) { + throw new UncheckedIOException("Could not append transaction to storage log", ex); + } + } + + @Override + public void flush() { + if (closed) + return; + if (logOut != null) { + try { + // flush the JVM buffer into the OS page cache; durable against a JVM process crash + logOut.flush(); + // in COMMIT mode also force the OS page cache to the device, so an acknowledged commit is durable + // against an OS crash or power loss. OS mode stops at the flush above and accepts that weaker guarantee. + if (syncMode == SyncMode.COMMIT) + logFos.getFD().sync(); + } catch (IOException ex) { + throw new UncheckedIOException("Could not flush storage log", ex); + } + } + } + + @Override + public void compact(final AbstractTinkerGraph graph) { + if (closed) + return; + // Write a fresh snapshot of the current committed state, then truncate the log. This must be crash-safe: at + // no point may a crash leave the store without a readable snapshot-or-log covering the committed state. + // Ordering is write-tmp -> fsync tmp -> atomically rename tmp over the snapshot -> fsync dir (the rename is + // now durable) -> delete the log -> fsync dir. The old snapshot is only ever replaced by an atomic rename, so + // a crash at any step leaves either the old (snapshot + log) or the new (snapshot) intact — never neither. + closeLog(); + ensureDirectory(); + final File tmp = new File(directory, SNAPSHOT_FILE + ".tmp"); + try (final FileOutputStream fos = new FileOutputStream(tmp); + final DataOutputStream out = new DataOutputStream(new BufferedOutputStream(fos))) { + writeHeader(out); + writeSnapshot(graph, out); + out.flush(); + // force the snapshot's bytes to the device before it is renamed into place + fos.getFD().sync(); + } catch (IOException ex) { + throw new UncheckedIOException("Could not write storage snapshot", ex); + } + + try { + // atomically replace the snapshot; no delete-then-rename window where the snapshot is briefly absent + atomicMove(tmp, snapshotFile); + // fsync the directory so the rename survives a crash before we touch the log + syncDirectory(); + + // truncate the log now that the snapshot durably reflects the committed state + if (logFile.exists() && !logFile.delete()) + throw new IOException("Could not truncate storage log " + logFile); + // fsync the directory again so the log's removal is durable + syncDirectory(); + } catch (IOException ex) { + throw new UncheckedIOException("Could not finalize storage snapshot", ex); + } + + // the log is now empty; the accumulated state lives in the snapshot + logBytesSinceCompaction = 0; + } + + @Override + public void maybeCompact(final AbstractTinkerGraph graph) { + if (closed || compactThresholdBytes <= 0) + return; + if (logBytesSinceCompaction >= compactThresholdBytes) + compact(graph); + } + + @Override + public void close() { + closeLog(); + closed = true; + } + + // ----------------------------------------------------------------------------------------- version marker + + /** + * Read and validate the store-level version marker, or create it for a new store. This is the single source of + * truth for the store's format version: a marker naming an unsupported version, or bad magic, fails the open + * loudly rather than risking a misread. + */ + private void establishStoreVersion() { + final boolean storeHasData = snapshotFile.exists() || logFile.exists(); + if (!versionFile.exists()) { + if (storeHasData && FORMAT_VERSION != 1) + throw new IllegalStateException(String.format( + "Storage location %s has data but no version marker; cannot confirm it is format version %d", + directory, FORMAT_VERSION)); + writeStoreVersion(); + return; + } + try (final DataInputStream in = new DataInputStream(new BufferedInputStream(new FileInputStream(versionFile)))) { + final byte[] magic = new byte[MAGIC.length]; + readFully(in, magic); + if (!Arrays.equals(magic, MAGIC)) + throw new IOException(String.format("%s is not a TinkerGraph storage version marker (bad magic)", versionFile)); + final byte version = in.readByte(); + if (version != FORMAT_VERSION) + throw new IOException(String.format( + "Unsupported storage format version %d at %s (this build writes %d); export via g.io() before upgrading", + version, directory, FORMAT_VERSION)); + } catch (IOException ex) { + throw new UncheckedIOException(String.format("Could not read storage version marker %s", versionFile), ex); + } + } + + private void writeStoreVersion() { + try (final FileOutputStream fos = new FileOutputStream(versionFile); + final DataOutputStream out = new DataOutputStream(fos)) { + out.write(MAGIC); + out.writeByte(FORMAT_VERSION); + out.flush(); + fos.getFD().sync(); + } catch (IOException ex) { + throw new UncheckedIOException(String.format("Could not write storage version marker %s", versionFile), ex); + } + } + + private void ensureDirectory() { + if (directory.exists()) { + if (!directory.isDirectory()) + throw new IllegalStateException(String.format("Storage location %s exists but is not a directory", directory)); + } else if (!directory.mkdirs()) { + throw new IllegalStateException(String.format("Could not create storage directory %s", directory)); + } + } + + // ----------------------------------------------------------------------------------------- fold / framing + + private void foldRecords(final File file, final Map vertices, final Map edges) { + final long fileLength = file.length(); + try (final DataInputStream in = new DataInputStream(new BufferedInputStream(new FileInputStream(file)))) { + long remaining = readAndVerifyHeader(in, file, fileLength); + while (true) { + final byte[] record = readFrame(in, remaining); + if (record == null) + break; + remaining -= 2L * Integer.BYTES + record.length; + decodeFrame(record, vertices, edges); + } + } catch (IOException ex) { + throw new UncheckedIOException(String.format("Could not read storage file %s", file), ex); + } + } + + /** + * Read and validate the per-file header (magic only), returning the number of record bytes that follow it. + */ + private long readAndVerifyHeader(final DataInputStream in, final File file, final long fileLength) throws IOException { + if (fileLength == 0) + return 0; + if (fileLength < HEADER_SIZE) + throw new IOException(String.format("Corrupt storage file %s: shorter than its %d-byte header", file, HEADER_SIZE)); + final byte[] magic = new byte[MAGIC.length]; + readFully(in, magic); + if (!Arrays.equals(magic, MAGIC)) + throw new IOException(String.format("%s is not a TinkerGraph storage file (bad magic)", file)); + return fileLength - HEADER_SIZE; + } + + private void ensureLogOpen() { + if (logOut == null) { + try { + final boolean freshFile = !logFile.exists() || logFile.length() == 0; + // retain the FileOutputStream so flush() can reach its FileDescriptor for fsync + logFos = new FileOutputStream(logFile, true); + logOut = new DataOutputStream(new BufferedOutputStream(logFos)); + if (freshFile) + writeHeader(logOut); + } catch (IOException ex) { + throw new UncheckedIOException("Could not open storage log for append", ex); + } + } + } + + private void closeLog() { + if (logOut != null) { + try { + logOut.flush(); + logOut.close(); + } catch (IOException ex) { + throw new UncheckedIOException("Could not close storage log", ex); + } finally { + logOut = null; + logFos = null; + } + } + } + + /** + * Write the per-file header ({@link #MAGIC}) at the start of a storage file. + */ + private static void writeHeader(final DataOutputStream out) throws IOException { + out.write(MAGIC); + } + + /** + * Write a framed record: a 4-byte big-endian payload length, a 4-byte CRC32 of the payload, then the payload. + * The checksum lets a reader tell a bit-flip inside a complete frame (corruption) from a short final frame left + * by an interrupted append (truncation). Available to codec subclasses writing per-element snapshot frames. + */ + protected static void writeFrame(final DataOutputStream out, final byte[] payload) throws IOException { + final CRC32 crc = new CRC32(); + crc.update(payload); + out.writeInt(payload.length); + out.writeInt((int) crc.getValue()); + out.write(payload); + } + + /** + * Read a framed record, or return {@code null} at end of the readable log. A frame only partially present is + * treated as an interrupted trailing append (truncation) and ends reading; a fully-present frame whose stored CRC + * does not match is genuine corruption and is raised. + */ + private static byte[] readFrame(final DataInputStream in, final long remaining) throws IOException { + if (remaining == 0) + return null; // clean end of file, exactly on a frame boundary + if (remaining < 2L * Integer.BYTES) + return null; // not even a full header left — interrupted append + + final int length = in.readInt(); + final int storedCrc = in.readInt(); + if (length < 0) + throw new IOException("Corrupt storage frame: negative payload length " + length); + if ((long) length > remaining - 2L * Integer.BYTES) + return null; // frame claims more bytes than remain — truncated trailing append + + final byte[] payload = new byte[length]; + try { + readFully(in, payload); + } catch (EOFException eof) { + return null; // partial trailing payload from an interrupted append + } + + final CRC32 crc = new CRC32(); + crc.update(payload); + if ((int) crc.getValue() != storedCrc) + throw new IOException(String.format( + "Corrupt storage frame: CRC mismatch (stored %08x, computed %08x) in a fully-present %d-byte record", + storedCrc, (int) crc.getValue(), length)); + return payload; + } + + private static void readFully(final InputStream in, final byte[] dst) throws IOException { + int off = 0; + while (off < dst.length) { + final int read = in.read(dst, off, dst.length - off); + if (read < 0) + throw new EOFException(); + off += read; + } + } + + /** + * Atomically move {@code source} onto {@code target}, replacing any existing target. Falls back to a non-atomic + * replacing move on filesystems that do not support atomic moves. + */ + private static void atomicMove(final File source, final File target) throws IOException { + try { + Files.move(source.toPath(), target.toPath(), + StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING); + } catch (AtomicMoveNotSupportedException anse) { + Files.move(source.toPath(), target.toPath(), StandardCopyOption.REPLACE_EXISTING); + } + } + + /** + * fsync the storage directory so that recent namespace changes (a rename into place, a file deletion) are durable. + */ + private void syncDirectory() { + try (final FileChannel dirChannel = FileChannel.open(directory.toPath(), StandardOpenOption.READ)) { + dirChannel.force(true); + } catch (IOException ex) { + // some platforms (notably Windows) cannot open a directory as a channel; the atomic rename is the + // durability guarantee there, so treat inability to sync the directory as non-fatal + } + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DefaultStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DefaultStorage.java new file mode 100644 index 00000000000..49c4a2becf4 --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DefaultStorage.java @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import java.util.function.Supplier; + +/** + * The built-in {@link TinkerStorage} engines that can be selected by name via the + * {@code gremlin.tinkergraph.storage} configuration key. A fully-qualified class name may be used instead of one of + * these names to plug in a custom engine. + */ +public enum DefaultStorage implements Supplier { + + /** + * A durable, append-only commit log ("write-ahead log") that serializes each committed transaction with + * GraphBinary. See {@link GraphBinaryStorage}. + */ + GRAPHBINARY { + @Override + public TinkerStorage get() { + return new GraphBinaryStorage(); + } + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DirectoryLock.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DirectoryLock.java new file mode 100644 index 00000000000..061e4e33728 --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DirectoryLock.java @@ -0,0 +1,115 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.channels.FileChannel; +import java.nio.channels.FileLock; +import java.nio.channels.OverlappingFileLockException; +import java.nio.file.StandardOpenOption; + +/** + * An exclusive, whole-process lock on a {@code TinkerStorageGraph} storage directory. A persistent, transactional + * TinkerGraph is a single-writer embedded store: two graphs opened on the same directory — whether in one JVM or + * across processes — would both append to and compact the same files, corrupting them. This holds an OS advisory lock + * ({@link FileLock}) on a {@code LOCK} file in the directory for the lifetime of the graph, so a second open fails + * fast rather than silently corrupting data. + *

+ * An OS lock (rather than a mere marker file) is used so the kernel releases it automatically if the JVM dies, + * avoiding a stale lock that would wedge the store after a crash. + *

+ * Note: {@link FileLock} semantics are unreliable on some network filesystems (notably NFS); this guarantee holds on + * local filesystems. + */ +public final class DirectoryLock implements AutoCloseable { + + static final String LOCK_FILE = "LOCK"; + + private final FileChannel channel; + private final FileLock lock; + private final File lockFile; + + private DirectoryLock(final FileChannel channel, final FileLock lock, final File lockFile) { + this.channel = channel; + this.lock = lock; + this.lockFile = lockFile; + } + + /** + * Acquire an exclusive lock on the {@code LOCK} file within {@code directory}. + * + * @param directory the storage directory, which must already exist + * @return the held lock, released by {@link #close()} + * @throws IllegalStateException if another graph (in this or another process) already holds the lock + */ + public static DirectoryLock acquire(final File directory) { + final File lockFile = new File(directory, LOCK_FILE); + FileChannel channel = null; + try { + channel = FileChannel.open(lockFile.toPath(), + StandardOpenOption.CREATE, StandardOpenOption.WRITE); + final FileLock lock = channel.tryLock(); + if (lock == null) { + channel.close(); + throw lockedByAnother(directory, null); + } + return new DirectoryLock(channel, lock, lockFile); + } catch (OverlappingFileLockException ofle) { + // another graph in *this* JVM already holds (or is acquiring) the lock on this file + closeQuietly(channel); + throw lockedByAnother(directory, ofle); + } catch (IOException ex) { + closeQuietly(channel); + throw new UncheckedIOException(String.format("Could not acquire storage lock for %s", directory), ex); + } + } + + private static IllegalStateException lockedByAnother(final File directory, final Throwable cause) { + return new IllegalStateException(String.format( + "Storage location %s is already in use by another TinkerStorageGraph (in this or another process); " + + "a persistent TinkerStorageGraph allows only a single writer", directory), cause); + } + + private static void closeQuietly(final FileChannel channel) { + if (channel != null) { + try { + channel.close(); + } catch (IOException ignored) { + // best effort on the failure path + } + } + } + + /** + * Release the lock and close the channel. Idempotent-friendly: safe to call once per acquired lock. + */ + @Override + public void close() { + try { + if (lock.isValid()) + lock.release(); + } catch (IOException ex) { + throw new UncheckedIOException(String.format("Could not release storage lock %s", lockFile), ex); + } finally { + closeQuietly(channel); + } + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java new file mode 100644 index 00000000000..e1404c3806e --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java @@ -0,0 +1,474 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.Edge; +import org.apache.tinkerpop.gremlin.structure.Property; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.structure.VertexProperty; +import org.apache.tinkerpop.gremlin.structure.io.binary.DataType; +import org.apache.tinkerpop.gremlin.structure.io.binary.GraphBinaryReader; +import org.apache.tinkerpop.gremlin.structure.io.binary.GraphBinaryWriter; +import org.apache.tinkerpop.gremlin.structure.io.binary.TypeSerializer; +import org.apache.tinkerpop.gremlin.structure.io.binary.TypeSerializerRegistry; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedProperty; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertexProperty; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.AbstractTinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerEdge; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex; + +import java.io.DataOutputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** + * The GraphBinary {@link TinkerStorage} codec on top of {@link AbstractLogStorage}. The base owns the durable + * log-structured machinery (file layout, {@code VERSION} marker, CRC framing, replay fold, {@link SyncMode} + * durability, crash-safe and threshold compaction); this class supplies only how an element is encoded and decoded. + *

+ * Rather than serialize a whole {@code DetachedVertex}/{@code DetachedEdge} (which repeats every property key and + * label as a full string on every element and wraps each property in a {@code VertexProperty} envelope), this codec + * writes element components directly: element ids and property values go through GraphBinary's scalar + * serializers (a one-byte {@link DataType} tag plus the raw value), while labels, property keys, and meta-property + * keys are dictionary-encoded to small integer refs. The dictionary is a single dense namespace built by first + * appearance; new entries are emitted as {@code OP_DICT_APPEND} records inside the same frame, before the entries + * that reference them, so a torn trailing frame drops a ref and its user atomically. + *

+ * Vertex-property ids are auto-generated and, by default, not persisted (they are regenerated on load); element and + * edge ids are always preserved. The snapshot streams one element per frame, so compaction never holds a second full + * copy of the graph in heap. + */ +public final class GraphBinaryStorage extends AbstractLogStorage { + + private static final byte OP_PUT_VERTEX = 1; + private static final byte OP_DEL_VERTEX = 2; + private static final byte OP_PUT_EDGE = 3; + private static final byte OP_DEL_EDGE = 4; + private static final byte OP_DICT_APPEND = 5; + + private final TypeSerializerRegistry registry = TypeSerializerRegistry.INSTANCE; + private final GraphBinaryWriter writer = new GraphBinaryWriter(registry); + private final GraphBinaryReader reader = new GraphBinaryReader(registry); + + /** + * Shared string dictionary for labels, property keys, and meta-property keys. Dense ids assigned by first + * appearance; grows monotonically within a write session and is rebuilt fresh on compaction and on replay. + */ + private final Map keyToId = new HashMap<>(); + private final List idToKey = new ArrayList<>(); + + /** + * When true, persist auto-generated vertex-property ids so they are stable across reopen. Written per vertex + * record so a store reopens correctly regardless of the reader's setting. + */ + private boolean preserveVertexPropertyIds = false; + + @Override + protected void configureCodec(final Configuration config) { + this.preserveVertexPropertyIds = config.getBoolean(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS, false); + } + + @Override + protected void beginReplay() { + keyToId.clear(); + idToKey.clear(); + } + + // --------------------------------------------------------------------------------------------- encode + + @Override + protected byte[] encodeCommit(final long txVersion, + final Collection> changedVertices, + final Collection> changedEdges) throws IOException { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + // register the strings introduced by this commit (deletes carry only an id, no strings) + final List appends = new ArrayList<>(); + for (final TinkerStorageMutation m : changedVertices) + if (!m.isDeleted()) registerVertexStrings(m.element(), appends); + for (final TinkerStorageMutation m : changedEdges) + if (!m.isDeleted()) registerEdgeStrings(m.element(), appends); + + writeVarInt(buf, appends.size() + changedVertices.size() + changedEdges.size()); + // dictionary appends first, so every ref below resolves during the fold + for (final String s : appends) { + buf.writeByte(OP_DICT_APPEND); + writeVarInt(buf, keyToId.get(s)); + writeString(buf, s); + } + for (final TinkerStorageMutation m : changedVertices) { + if (m.isDeleted()) { + buf.writeByte(OP_DEL_VERTEX); + writeScalar(buf, m.id()); + } else { + buf.writeByte(OP_PUT_VERTEX); + writeVertexRecord(buf, m.element()); + } + } + for (final TinkerStorageMutation m : changedEdges) { + if (m.isDeleted()) { + buf.writeByte(OP_DEL_EDGE); + writeScalar(buf, m.id()); + } else { + buf.writeByte(OP_PUT_EDGE); + writeEdgeRecord(buf, m.element()); + } + } + return buf.toWrittenArray(); + } + + @Override + protected void writeSnapshot(final AbstractTinkerGraph graph, final DataOutputStream out) throws IOException { + // Preserve the existing dictionary numbering rather than renumbering: the compaction crash window can leave + // the new snapshot in place with the old log not yet truncated, and that log's refs use the current + // numbering. Register any not-yet-seen live strings (this only extends the dictionary, never renumbers), then + // emit the whole dictionary as a self-contained header frame so a snapshot-only replay resolves every ref. + // committed*() rather than vertices()/edges(): the snapshot must reflect only committed state, never the + // calling thread's uncommitted transaction-local mutations, which the transaction-aware iterators would expose. + final List ignored = new ArrayList<>(); + Iterator vertices = graph.committedVertices(); + while (vertices.hasNext()) + registerVertexStrings(vertices.next(), ignored); + Iterator edges = graph.committedEdges(); + while (edges.hasNext()) + registerEdgeStrings(edges.next(), ignored); + + final TinkerByteBuffer dictBuf = new TinkerByteBuffer(); + writeVarInt(dictBuf, idToKey.size()); + for (int id = 0; id < idToKey.size(); id++) { + dictBuf.writeByte(OP_DICT_APPEND); + writeVarInt(dictBuf, id); + writeString(dictBuf, idToKey.get(id)); + } + writeFrame(out, dictBuf.toWrittenArray()); + + // one element per frame; all keys are already in the dictionary header, so no per-frame appends + vertices = graph.committedVertices(); + while (vertices.hasNext()) + writeElementFrame(out, OP_PUT_VERTEX, vertices.next()); + edges = graph.committedEdges(); + while (edges.hasNext()) + writeElementFrame(out, OP_PUT_EDGE, edges.next()); + } + + /** + * Write one element as its own single-entry put frame. One element is held in memory at a time, so a large graph + * is never buffered whole. + */ + private void writeElementFrame(final DataOutputStream out, final byte op, final Object element) throws IOException { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + writeVarInt(buf, 1); + buf.writeByte(op); + if (op == OP_PUT_VERTEX) writeVertexRecord(buf, (Vertex) element); + else writeEdgeRecord(buf, (Edge) element); + writeFrame(out, buf.toWrittenArray()); + } + + private void registerVertexStrings(final Vertex v, final List appends) { + for (final String label : v.labels()) + register(label, appends); + final Iterator> vps = v.properties(); + while (vps.hasNext()) { + final VertexProperty vp = vps.next(); + register(vp.key(), appends); + final Iterator> metas = vp.properties(); + while (metas.hasNext()) + register(metas.next().key(), appends); + } + } + + private void registerEdgeStrings(final Edge e, final List appends) { + register(e.label(), appends); + final Iterator> props = e.properties(); + while (props.hasNext()) + register(props.next().key(), appends); + } + + private void register(final String s, final List appends) { + if (!keyToId.containsKey(s)) { + final int id = idToKey.size(); + keyToId.put(s, id); + idToKey.add(s); + appends.add(s); + } + } + + private void writeVertexRecord(final TinkerByteBuffer buf, final Vertex v) throws IOException { + writeScalar(buf, v.id()); + final Set labels = v.labels(); + writeVarInt(buf, labels.size()); + for (final String label : labels) + writeVarInt(buf, keyToId.get(label)); + + // self-describing flag: whether each value below carries a persisted vertex-property id + buf.writeByte(preserveVertexPropertyIds ? 1 : 0); + + // group vertex properties by key so multi-properties (list/set) round-trip + final Map>> groups = new LinkedHashMap<>(); + final Iterator> vps = v.properties(); + while (vps.hasNext()) { + final VertexProperty vp = vps.next(); + groups.computeIfAbsent(vp.key(), k -> new ArrayList<>()).add(vp); + } + writeVarInt(buf, groups.size()); + for (final Map.Entry>> group : groups.entrySet()) { + writeVarInt(buf, keyToId.get(group.getKey())); + final List> values = group.getValue(); + writeVarInt(buf, values.size()); + for (final VertexProperty vp : values) { + writeScalar(buf, vp.value()); + if (preserveVertexPropertyIds) + writeScalar(buf, vp.id()); + final List> metas = new ArrayList<>(); + vp.properties().forEachRemaining(metas::add); + writeVarInt(buf, metas.size()); + for (final Property meta : metas) { + writeVarInt(buf, keyToId.get(meta.key())); + writeScalar(buf, meta.value()); + } + } + } + } + + private void writeEdgeRecord(final TinkerByteBuffer buf, final Edge e) throws IOException { + writeScalar(buf, e.id()); + writeVarInt(buf, keyToId.get(e.label())); + writeScalar(buf, e.outVertex().id()); + writeScalar(buf, e.inVertex().id()); + final List> props = new ArrayList<>(); + e.properties().forEachRemaining(props::add); + writeVarInt(buf, props.size()); + for (final Property p : props) { + writeVarInt(buf, keyToId.get(p.key())); + writeScalar(buf, p.value()); + } + } + + // --------------------------------------------------------------------------------------------- decode + + @Override + protected void decodeFrame(final byte[] record, + final Map vertices, + final Map edges) throws IOException { + final TinkerByteBuffer buf = new TinkerByteBuffer(record); + final int entryCount = readVarInt(buf); + for (int i = 0; i < entryCount; i++) { + final byte op = buf.readByte(); + switch (op) { + case OP_DICT_APPEND: { + final int id = readVarInt(buf); + final String s = readString(buf); + // idempotent: a snapshot header defines the whole dictionary, and a log surviving the compaction + // crash window may re-append entries the snapshot already established. Re-appending an existing + // id with the same string is a no-op; a mismatch or a gap is corruption. + if (id < idToKey.size()) { + if (!idToKey.get(id).equals(s)) + throw new IOException(String.format("Corrupt storage: dictionary id %d redefined ('%s' vs '%s')", id, idToKey.get(id), s)); + } else if (id == idToKey.size()) { + idToKey.add(s); + } else { + throw new IOException(String.format("Corrupt storage: dictionary append gap (got %d, expected <= %d)", id, idToKey.size())); + } + // rebuild the write-side mapping too, first appearance wins, so a write session after this replay + // resumes the existing numbering instead of re-appending every live key as a duplicate on each reopen + keyToId.putIfAbsent(s, id); + break; + } + case OP_PUT_VERTEX: { + final DetachedVertex v = readVertexRecord(buf); + vertices.put(v.id(), v); + break; + } + case OP_DEL_VERTEX: { + vertices.remove(readScalar(buf)); + break; + } + case OP_PUT_EDGE: { + final DetachedEdge e = readEdgeRecord(buf); + edges.put(e.id(), e); + break; + } + case OP_DEL_EDGE: { + edges.remove(readScalar(buf)); + break; + } + default: + throw new IOException("Unknown storage op code: " + op); + } + } + } + + private DetachedVertex readVertexRecord(final TinkerByteBuffer buf) throws IOException { + final Object id = readScalar(buf); + final DetachedVertex.Builder b = DetachedVertex.build().setId(id); + final int labelCount = readVarInt(buf); + if (labelCount == 1) { + b.setLabel(resolveKey(buf)); + } else if (labelCount > 1) { + final Set labels = new LinkedHashSet<>(); + for (int i = 0; i < labelCount; i++) + labels.add(resolveKey(buf)); + b.setLabels(labels); + } + final boolean hasVpIds = buf.readByte() != 0; + final int keyGroupCount = readVarInt(buf); + for (int g = 0; g < keyGroupCount; g++) { + final String key = resolveKey(buf); + final int valueCount = readVarInt(buf); + for (int j = 0; j < valueCount; j++) { + final Object value = readScalar(buf); + final DetachedVertexProperty.Builder vpb = DetachedVertexProperty.build().setLabel(key).setValue(value); + if (hasVpIds) + vpb.setId(readScalar(buf)); + final int metaCount = readVarInt(buf); + for (int m = 0; m < metaCount; m++) { + final String metaKey = resolveKey(buf); + final Object metaValue = readScalar(buf); + vpb.addProperty(new DetachedProperty<>(metaKey, metaValue)); + } + b.addProperty(vpb.create()); + } + } + return b.create(); + } + + private DetachedEdge readEdgeRecord(final TinkerByteBuffer buf) throws IOException { + final Object id = readScalar(buf); + final String label = resolveKey(buf); + final Object outVId = readScalar(buf); + final Object inVId = readScalar(buf); + final DetachedEdge.Builder b = DetachedEdge.build().setId(id).setLabel(label) + .setOutV(DetachedVertex.build().setId(outVId).create()) + .setInV(DetachedVertex.build().setId(inVId).create()); + final int propCount = readVarInt(buf); + for (int i = 0; i < propCount; i++) { + final String key = resolveKey(buf); + final Object value = readScalar(buf); + b.addProperty(new DetachedProperty<>(key, value)); + } + return b.create(); + } + + // --------------------------------------------------------------------------------------------- primitives + + /** + * Write a value as a one-byte {@link DataType} tag followed by the raw value (no value-flag byte). A {@code null} + * is a single {@link DataType#UNSPECIFIED_NULL} tag. + */ + @SuppressWarnings({"unchecked", "rawtypes"}) + private void writeScalar(final TinkerByteBuffer buf, final Object value) throws IOException { + if (value == null) { + buf.writeByte(DataType.UNSPECIFIED_NULL.getCodeByte()); + return; + } + final TypeSerializer serializer = registry.getSerializer(value.getClass()); + buf.writeByte(serializer.getDataType().getCodeByte()); + serializer.writeValue(value, buf, writer, false); + } + + @SuppressWarnings({"unchecked", "rawtypes"}) + private Object readScalar(final TinkerByteBuffer buf) throws IOException { + final int code = Byte.toUnsignedInt(buf.readByte()); + final DataType dataType = DataType.get(code); + if (dataType == null) + throw new IOException(String.format("Corrupt storage: unknown value type code 0x%02X", code)); + if (dataType == DataType.UNSPECIFIED_NULL) + return null; + final TypeSerializer serializer = registry.getSerializer(dataType); + return serializer.readValue(buf, reader, false); + } + + private static void writeString(final TinkerByteBuffer buf, final String s) { + final byte[] bytes = s.getBytes(StandardCharsets.UTF_8); + writeVarInt(buf, bytes.length); + buf.writeBytes(bytes); + } + + /** + * Resolve the next dictionary ref in {@code buf} to its string. A ref that names an entry the dictionary does not + * hold is corruption, and is reported as such rather than raised as an {@code IndexOutOfBoundsException} from the + * backing list. + */ + private String resolveKey(final TinkerByteBuffer buf) throws IOException { + final int id = readVarInt(buf); + if (id >= idToKey.size()) + throw new IOException(String.format( + "Corrupt storage frame: dictionary ref %d with only %d entries defined", id, idToKey.size())); + return idToKey.get(id); + } + + private static String readString(final TinkerByteBuffer buf) throws IOException { + final int length = readVarInt(buf); + // check the declared length against what the frame actually holds before allocating. The frame itself is + // already bounded against the file by AbstractLogStorage.readFrame, but a length inside the frame is not, + // so an unchecked allocation here would let a small corrupt record demand gigabytes and raise + // OutOfMemoryError instead of the IOException a corrupt frame is contracted to produce. + if (length > buf.readableBytes()) + throw new IOException(String.format( + "Corrupt storage frame: string of %d bytes declared with only %d readable in the record", + length, buf.readableBytes())); + final byte[] bytes = new byte[length]; + buf.readBytes(bytes); + return new String(bytes, StandardCharsets.UTF_8); + } + + /** + * Unsigned LEB128 varint. Counts and dictionary refs are small and non-negative, so they cost one byte in the + * common case. + */ + private static void writeVarInt(final TinkerByteBuffer buf, final int value) { + int v = value; + while ((v & ~0x7F) != 0) { + buf.writeByte((v & 0x7F) | 0x80); + v >>>= 7; + } + buf.writeByte(v & 0x7F); + } + + private static int readVarInt(final TinkerByteBuffer buf) throws IOException { + int result = 0; + int shift = 0; + byte b; + do { + // Java masks a shift count to five bits, so without this bound an over-long encoding wraps around and + // yields an arbitrary (possibly negative) value rather than failing. Every count, length and dictionary + // ref in the format is non-negative, so anything that does not fit in five groups is corruption. + if (shift >= Integer.SIZE) + throw new IOException("Corrupt storage frame: over-long varint encoding"); + b = buf.readByte(); + result |= (b & 0x7F) << shift; + shift += 7; + } while ((b & 0x80) != 0); + if (result < 0) + throw new IOException("Corrupt storage frame: negative varint value " + result); + return result; + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/IndexDefinitions.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/IndexDefinitions.java new file mode 100644 index 00000000000..07e63b9bd60 --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/IndexDefinitions.java @@ -0,0 +1,205 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.OutputStreamWriter; +import java.io.Writer; +import java.nio.charset.StandardCharsets; +import java.nio.channels.FileChannel; +import java.nio.file.AtomicMoveNotSupportedException; +import java.nio.file.Files; +import java.nio.file.StandardCopyOption; +import java.nio.file.StandardOpenOption; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Set; + +/** + * The set of property keys a persistent graph indexes, held in a small file beside the storage engine's own files. + *

+ * Index definitions are deliberately kept out of the transactional log. An index is purely an optimization, so losing + * a definition to a crash costs a rebuild rather than any data: a crash between {@code createIndex} and the write here + * drops a definition the caller can recreate, and one after {@code dropIndex} resurrects an index that is dropped + * again. Neither can change a query result, which is what makes a file outside the write-ahead log an honest place to + * keep this. Nothing that affects correctness may be stored this way. + *

+ * The file is line oriented and readable, one record per line: {@code V} or {@code E}, a tab, then the property key + * with backslash, tab, newline and carriage return escaped, so a key containing any of them survives a round trip. + */ +public final class IndexDefinitions { + + private static final Logger logger = LoggerFactory.getLogger(IndexDefinitions.class); + + static final String INDEX_FILE = "INDEXES"; + + private static final String VERTEX = "V"; + private static final String EDGE = "E"; + + private final Set vertexKeys; + private final Set edgeKeys; + + public IndexDefinitions(final Set vertexKeys, final Set edgeKeys) { + this.vertexKeys = new LinkedHashSet<>(vertexKeys); + this.edgeKeys = new LinkedHashSet<>(edgeKeys); + } + + public Set vertexKeys() { + return vertexKeys; + } + + public Set edgeKeys() { + return edgeKeys; + } + + public boolean isEmpty() { + return vertexKeys.isEmpty() && edgeKeys.isEmpty(); + } + + /** + * Read the definitions recorded in {@code directory}, or an empty set if none have been recorded. + *

+ * A file that cannot be read is reported and treated as empty rather than raised. The graph then opens with no + * indexes, which is exactly how it behaved before definitions were recorded at all, so an unreadable file can + * never make a store less openable than the data it holds. + */ + public static IndexDefinitions read(final File directory) { + final File file = new File(directory, INDEX_FILE); + if (!file.isFile()) + return new IndexDefinitions(new LinkedHashSet<>(), new LinkedHashSet<>()); + + final Set vertexKeys = new LinkedHashSet<>(); + final Set edgeKeys = new LinkedHashSet<>(); + try { + final List lines = Files.readAllLines(file.toPath(), StandardCharsets.UTF_8); + for (final String line : lines) { + if (line.isEmpty() || line.charAt(0) == '#') + continue; + final int tab = line.indexOf('\t'); + if (tab < 0) + throw new IOException("Malformed index definition line: " + line); + final String key = unescape(line.substring(tab + 1)); + switch (line.substring(0, tab)) { + case VERTEX: vertexKeys.add(key); break; + case EDGE: edgeKeys.add(key); break; + default: throw new IOException("Unknown index element type in line: " + line); + } + } + } catch (IOException ex) { + logger.warn("Could not read index definitions from {}; opening with no indexes. " + + "Recreate them with createIndex() if they are wanted.", file, ex); + return new IndexDefinitions(new LinkedHashSet<>(), new LinkedHashSet<>()); + } + return new IndexDefinitions(vertexKeys, edgeKeys); + } + + /** + * Replace the definitions recorded in {@code directory}. Written to a temporary file, forced to the device and + * renamed into place, so a crash leaves either the previous set or the new one and never a partial file. + */ + public void write(final File directory) { + final File file = new File(directory, INDEX_FILE); + if (isEmpty()) { + try { + Files.deleteIfExists(file.toPath()); + syncDirectory(directory); + } catch (IOException ex) { + logger.warn("Could not remove index definitions file {}", file, ex); + } + return; + } + + final File tmp = new File(directory, INDEX_FILE + ".tmp"); + try { + try (final FileOutputStream fos = new FileOutputStream(tmp); + final Writer out = new OutputStreamWriter(fos, StandardCharsets.UTF_8)) { + out.write("# TinkerGraph index definitions; recreated on open\n"); + for (final String key : vertexKeys) + out.write(VERTEX + '\t' + escape(key) + '\n'); + for (final String key : edgeKeys) + out.write(EDGE + '\t' + escape(key) + '\n'); + out.flush(); + fos.getFD().sync(); + } + atomicMove(tmp, file); + syncDirectory(directory); + } catch (IOException ex) { + logger.warn("Could not record index definitions in {}; they will not survive a reopen", file, ex); + } + } + + private static String escape(final String key) { + final StringBuilder sb = new StringBuilder(key.length()); + for (int i = 0; i < key.length(); i++) { + final char c = key.charAt(i); + switch (c) { + case '\\': sb.append("\\\\"); break; + case '\t': sb.append("\\t"); break; + case '\n': sb.append("\\n"); break; + case '\r': sb.append("\\r"); break; + default: sb.append(c); + } + } + return sb.toString(); + } + + private static String unescape(final String value) throws IOException { + final StringBuilder sb = new StringBuilder(value.length()); + for (int i = 0; i < value.length(); i++) { + final char c = value.charAt(i); + if (c != '\\') { + sb.append(c); + continue; + } + if (++i == value.length()) + throw new IOException("Index definition ends with a dangling escape: " + value); + switch (value.charAt(i)) { + case '\\': sb.append('\\'); break; + case 't': sb.append('\t'); break; + case 'n': sb.append('\n'); break; + case 'r': sb.append('\r'); break; + default: throw new IOException("Unknown escape in index definition: " + value); + } + } + return sb.toString(); + } + + private static void atomicMove(final File source, final File target) throws IOException { + try { + Files.move(source.toPath(), target.toPath(), + StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING); + } catch (AtomicMoveNotSupportedException anse) { + Files.move(source.toPath(), target.toPath(), StandardCopyOption.REPLACE_EXISTING); + } + } + + private static void syncDirectory(final File directory) { + try (final FileChannel dirChannel = FileChannel.open(directory.toPath(), StandardOpenOption.READ)) { + dirChannel.force(true); + } catch (IOException ex) { + // some platforms (notably Windows) cannot open a directory as a channel; the atomic rename is the + // durability guarantee there, so treat inability to sync the directory as non-fatal + } + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/SyncMode.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/SyncMode.java new file mode 100644 index 00000000000..d83e2d00a9b --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/SyncMode.java @@ -0,0 +1,75 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +/** + * The durability mode a {@link TinkerStorage} engine applies when flushing a committed transaction to disk, selected + * with the {@code gremlin.tinkergraph.storage.sync} configuration key. Each value is a complete, mutually-exclusive + * choice; they are ordered from strongest to weakest durability. + *

+ * The name of a mode describes when data is made durable, not what action is taken: {@link #COMMIT} + * performs an {@code fsync} so acknowledged commits survive an OS crash or power loss, whereas {@link #OS} only pushes + * bytes into the operating system's page cache, so commits survive a crash of the JVM process but not of the OS. + */ +public enum SyncMode { + + /** + * {@code fsync} on every commit. An acknowledged commit is durable against process crash, OS crash, and power + * loss. This is the default and the mode that honors the "each committed transaction is durably written to disk" + * contract. + */ + COMMIT, + + /** + * Flush to the operating system on every commit, but do not {@code fsync}. An acknowledged commit survives a crash + * of the JVM process but may be lost on an OS crash or power loss, since the data can still be sitting in the OS + * page cache. Faster than {@link #COMMIT}; use only when that weaker guarantee is acceptable. + */ + OS; + + // TODO: add an INTERVAL mode (group commit) — a peer value on this same key, encoded as "interval:", that + // fsyncs at most once per window rather than once per commit, bounding the crash-loss window by time while + // amortizing fsync cost across commits. It implies fsync (a batched COMMIT), so it slots in as a third + // mutually-exclusive mode without changing the meaning of COMMIT or OS. + // + // This is coupled to the commit-write lock that serializes concurrent commits (see the persist/flush step in + // TinkerTransaction.doCommit): once that lock exists, COMMIT holds it across the fsync, so every commit serializes + // on disk-sync latency. INTERVAL is the fix — hold the lock only for the buffer append (fast) and fsync one batch + // for many transactions. So INTERVAL should be built on top of that lock, not before it: it is the performance + // pass that makes serialized commits cheap, which is why it is deferred until concurrent commits are serialized. + + /** + * Resolve a configuration value to a {@link SyncMode}, matched case-insensitively, defaulting to {@link #COMMIT} + * when unset. + * + * @param configValue the raw configuration value, or {@code null} when unset + * @return the resolved mode + * @throws IllegalArgumentException if the value does not name a known mode + */ + public static SyncMode fromConfigValue(final String configValue) { + if (null == configValue) + return COMMIT; + try { + return SyncMode.valueOf(configValue.trim().toUpperCase()); + } catch (IllegalArgumentException iae) { + throw new IllegalArgumentException(String.format( + "Unknown storage sync mode '%s'; valid values are 'commit' and 'os'", configValue), iae); + } + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerByteBuffer.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerByteBuffer.java new file mode 100644 index 00000000000..d32a212a307 --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerByteBuffer.java @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.tinkerpop.gremlin.structure.io.Buffer; + +import java.io.IOException; +import java.io.OutputStream; +import java.nio.ByteBuffer; +import java.util.Arrays; + +/** + * A self-contained, heap {@code byte[]}-backed implementation of the gremlin-core {@link Buffer} abstraction, used to + * drive {@code GraphBinaryWriter}/{@code GraphBinaryReader} without depending on gremlin-util's Netty-backed buffer. + * All multi-byte values are big-endian, matching the wire order the Netty implementation uses. The backing array grows + * as needed on write. This class is not thread-safe; a buffer is used by a single thread for a single + * serialize/deserialize. + */ +public final class TinkerByteBuffer implements Buffer { + + private static final int DEFAULT_CAPACITY = 256; + + private byte[] array; + private int readerIndex = 0; + private int writerIndex = 0; + private int markedWriterIndex = 0; + private int referenceCount = 1; + + public TinkerByteBuffer() { + this(DEFAULT_CAPACITY); + } + + public TinkerByteBuffer(final int initialCapacity) { + this.array = new byte[Math.max(initialCapacity, 1)]; + } + + /** + * Wraps an existing array for reading. The writer index is positioned at the end of the supplied data. + */ + public TinkerByteBuffer(final byte[] data) { + this.array = data; + this.writerIndex = data.length; + } + + /** + * Returns a copy of the readable-region bytes (from reader index to writer index). Does not change indexes. + */ + public byte[] toReadableArray() { + return Arrays.copyOfRange(array, readerIndex, writerIndex); + } + + /** + * Returns a copy of all written bytes (index 0 to writer index). Does not change indexes. + */ + public byte[] toWrittenArray() { + return Arrays.copyOfRange(array, 0, writerIndex); + } + + private void ensureWritable(final int additional) { + final int required = writerIndex + additional; + if (required <= array.length) + return; + int newCapacity = array.length; + while (newCapacity < required) + newCapacity <<= 1; + array = Arrays.copyOf(array, newCapacity); + } + + private void checkReadable(final int length) { + if (readerIndex + length > writerIndex) + throw new IndexOutOfBoundsException(String.format( + "Not enough readable bytes: need %d at index %d but writer index is %d", length, readerIndex, writerIndex)); + } + + @Override + public int readableBytes() { + return writerIndex - readerIndex; + } + + @Override + public int readerIndex() { + return readerIndex; + } + + @Override + public Buffer readerIndex(final int readerIndex) { + if (readerIndex < 0 || readerIndex > writerIndex) + throw new IndexOutOfBoundsException("readerIndex: " + readerIndex); + this.readerIndex = readerIndex; + return this; + } + + @Override + public int writerIndex() { + return writerIndex; + } + + @Override + public Buffer writerIndex(final int writerIndex) { + if (writerIndex < readerIndex) + throw new IndexOutOfBoundsException("writerIndex: " + writerIndex); + ensureWritable(writerIndex - this.writerIndex); + this.writerIndex = writerIndex; + return this; + } + + @Override + public Buffer markWriterIndex() { + this.markedWriterIndex = writerIndex; + return this; + } + + @Override + public Buffer resetWriterIndex() { + this.writerIndex = markedWriterIndex; + return this; + } + + @Override + public int capacity() { + return array.length; + } + + @Override + public boolean isDirect() { + return false; + } + + @Override + public boolean readBoolean() { + return readByte() != 0; + } + + @Override + public byte readByte() { + checkReadable(1); + return array[readerIndex++]; + } + + @Override + public short readShort() { + checkReadable(2); + return (short) (((array[readerIndex++] & 0xFF) << 8) | (array[readerIndex++] & 0xFF)); + } + + @Override + public int readInt() { + checkReadable(4); + return ((array[readerIndex++] & 0xFF) << 24) | + ((array[readerIndex++] & 0xFF) << 16) | + ((array[readerIndex++] & 0xFF) << 8) | + (array[readerIndex++] & 0xFF); + } + + @Override + public long readLong() { + checkReadable(8); + long value = 0; + for (int i = 0; i < 8; i++) + value = (value << 8) | (array[readerIndex++] & 0xFF); + return value; + } + + @Override + public float readFloat() { + return Float.intBitsToFloat(readInt()); + } + + @Override + public double readDouble() { + return Double.longBitsToDouble(readLong()); + } + + @Override + public Buffer readBytes(final byte[] destination) { + return readBytes(destination, 0, destination.length); + } + + @Override + public Buffer readBytes(final byte[] destination, final int dstIndex, final int length) { + checkReadable(length); + System.arraycopy(array, readerIndex, destination, dstIndex, length); + readerIndex += length; + return this; + } + + @Override + public Buffer readBytes(final ByteBuffer dst) { + final int length = dst.remaining(); + checkReadable(length); + dst.put(array, readerIndex, length); + readerIndex += length; + return this; + } + + @Override + public Buffer readBytes(final OutputStream out, final int length) throws IOException { + checkReadable(length); + out.write(array, readerIndex, length); + readerIndex += length; + return this; + } + + @Override + public Buffer writeBoolean(final boolean value) { + return writeByte(value ? 1 : 0); + } + + @Override + public Buffer writeByte(final int value) { + ensureWritable(1); + array[writerIndex++] = (byte) value; + return this; + } + + @Override + public Buffer writeShort(final int value) { + ensureWritable(2); + array[writerIndex++] = (byte) (value >>> 8); + array[writerIndex++] = (byte) value; + return this; + } + + @Override + public Buffer writeInt(final int value) { + ensureWritable(4); + array[writerIndex++] = (byte) (value >>> 24); + array[writerIndex++] = (byte) (value >>> 16); + array[writerIndex++] = (byte) (value >>> 8); + array[writerIndex++] = (byte) value; + return this; + } + + @Override + public Buffer writeLong(final long value) { + ensureWritable(8); + for (int i = 56; i >= 0; i -= 8) + array[writerIndex++] = (byte) (value >>> i); + return this; + } + + @Override + public Buffer writeFloat(final float value) { + return writeInt(Float.floatToIntBits(value)); + } + + @Override + public Buffer writeDouble(final double value) { + return writeLong(Double.doubleToLongBits(value)); + } + + @Override + public Buffer writeBytes(final byte[] src) { + return writeBytes(src, 0, src.length); + } + + @Override + public Buffer writeBytes(final ByteBuffer src) { + final int length = src.remaining(); + ensureWritable(length); + src.get(array, writerIndex, length); + writerIndex += length; + return this; + } + + @Override + public Buffer writeBytes(final byte[] src, final int srcIndex, final int length) { + ensureWritable(length); + System.arraycopy(src, srcIndex, array, writerIndex, length); + writerIndex += length; + return this; + } + + @Override + public boolean release() { + return --referenceCount <= 0; + } + + @Override + public Buffer retain() { + referenceCount++; + return this; + } + + @Override + public int referenceCount() { + return referenceCount; + } + + @Override + public int nioBufferCount() { + return 1; + } + + @Override + public ByteBuffer[] nioBuffers() { + return new ByteBuffer[] { nioBuffer() }; + } + + @Override + public ByteBuffer[] nioBuffers(final int index, final int length) { + return new ByteBuffer[] { nioBuffer(index, length) }; + } + + @Override + public ByteBuffer nioBuffer() { + return nioBuffer(readerIndex, readableBytes()); + } + + @Override + public ByteBuffer nioBuffer(final int index, final int length) { + return ByteBuffer.wrap(Arrays.copyOfRange(array, index, index + length)); + } + + @Override + public Buffer getBytes(final int index, final byte[] dst) { + System.arraycopy(array, index, dst, 0, dst.length); + return this; + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorage.java new file mode 100644 index 00000000000..0d90879c8f3 --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorage.java @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.AbstractTinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerEdge; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex; + +import java.util.Collection; + +/** + * A pluggable durable storage engine for a transactional {@code TinkerStorageGraph}. Implementations persist the + * changeset of each committed transaction to disk and rebuild the in-memory graph on open. The graph remains the + * authoritative in-memory copy (write-through); the engine is a durable mirror. + *

+ * The lifecycle is: + *

    + *
  1. {@link #open(AbstractTinkerGraph, Configuration)} — resolve the backing location and ready the store.
    2. + *
  2. {@link #replay(AbstractTinkerGraph)} — rebuild in-memory state from what was previously persisted.
    3. + *
  3. {@link #persist(long, Collection, Collection)} then {@link #flush()} — called on each transaction commit, + * before the in-memory state is committed, so a failure aborts the commit and leaves memory and disk consistent.
    4. + *
  4. {@link #compact(AbstractTinkerGraph)} — optionally fold accumulated changes into a compact snapshot.
    5. + *
  5. {@link #close()} — release resources.
    6. + *
+ * A new engine can be selected by name (see {@code TinkerStorageGraph.DefaultStorage}) or by fully-qualified class + * name via the {@code gremlin.tinkergraph.storage} configuration key. Implementations must provide a public no-argument + * constructor. + */ +public interface TinkerStorage extends AutoCloseable { + + /** + * Prepare the storage engine for use, resolving its backing location from the supplied configuration. Called once + * during graph construction before {@link #replay(AbstractTinkerGraph)}. + * + * @param graph the graph that owns this engine + * @param config the graph configuration, including {@code gremlin.tinkergraph.storage.directory} + */ + void open(AbstractTinkerGraph graph, Configuration config); + + /** + * Rebuild the in-memory state of the graph from previously persisted data. Called once during graph construction. + * Implementations should re-apply persisted elements through the graph's own mutation API; the graph sets its + * {@code loading} guard for the duration so this does not re-persist. + * + * @param graph the graph to populate + */ + void replay(AbstractTinkerGraph graph); + + /** + * Durably record the changeset of a committing transaction. Called from within the transaction commit, while the + * changed elements are locked, before the in-memory commit is applied. Each mutation is either a put (added or + * modified element) or a delete (see {@link TinkerStorageMutation}). + * + * @param txVersion the version number of the committing transaction + * @param changedVertices the vertex mutations in this transaction + * @param changedEdges the edge mutations in this transaction + */ + void persist(long txVersion, + Collection> changedVertices, + Collection> changedEdges); + + /** + * Force any buffered writes to durable storage. Called after {@link #persist(long, Collection, Collection)} as the + * commit's durability point. + */ + void flush(); + + /** + * Fold accumulated changes into a compact representation of the current committed state, reclaiming space. Safe to + * call at any time; typically invoked on {@link #close()} or on a size/commit-count threshold. + * + * @param graph the graph whose current committed state should be snapshotted + */ + void compact(AbstractTinkerGraph graph); + + /** + * Compact if the engine's accumulated data has grown past its own threshold, otherwise do nothing. Called on the + * commit path after {@link #flush()} (while the graph's commit lock is held) so a long-running graph that is never + * explicitly closed does not grow its backing storage without bound. The default is a no-op, leaving compaction + * entirely under the control of {@link #compact(AbstractTinkerGraph)} and {@link #close()}. + * + * @param graph the graph whose current committed state would be snapshotted + */ + default void maybeCompact(final AbstractTinkerGraph graph) { + // no-op by default; engines that accumulate an on-disk log override this to bound its growth + } + + /** + * {@inheritDoc} + *

+ * Flush and release resources. Does not delete persisted data. + */ + @Override + void close(); +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorageMutation.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorageMutation.java new file mode 100644 index 00000000000..726c6862fd1 --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorageMutation.java @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.tinkerpop.gremlin.structure.Element; + +/** + * A single element change within a committing transaction, handed to a {@link TinkerStorage} engine to persist. This + * is the stable, public view of a change that decouples storage engines from TinkerGraph's internal transactional + * containers. A mutation is either a put (the element was added or modified, {@link #element()} is non-null) + * or a delete ({@link #element()} is {@code null} and {@link #isDeleted()} is {@code true}). + * + * @param the element type ({@code TinkerVertex} or {@code TinkerEdge}) + */ +public final class TinkerStorageMutation { + + private final Object id; + private final T element; + + /** + * Create a mutation for the given element id. + * + * @param id the element identifier (never {@code null}) + * @param element the committed element for a put, or {@code null} for a delete + */ + public TinkerStorageMutation(final Object id, final T element) { + this.id = id; + this.element = element; + } + + /** + * The identifier of the changed element. + */ + public Object id() { + return id; + } + + /** + * The committed element to persist, or {@code null} when this mutation is a deletion. + */ + public T element() { + return element; + } + + /** + * Returns {@code true} when this mutation deletes the element rather than adding or modifying it. + */ + public boolean isDeleted() { + return element == null; + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphProvider.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphProvider.java index fe17fd7d71e..170b43f8f55 100644 --- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphProvider.java +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphProvider.java @@ -21,7 +21,6 @@ import org.apache.commons.configuration2.Configuration; import org.apache.tinkerpop.gremlin.AbstractGraphProvider; import org.apache.tinkerpop.gremlin.LoadGraphWith; -import org.apache.tinkerpop.gremlin.TestHelper; import org.apache.tinkerpop.gremlin.structure.Graph; import org.apache.tinkerpop.gremlin.structure.GraphTest; import org.apache.tinkerpop.gremlin.structure.VertexProperty; @@ -39,7 +38,6 @@ import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertexProperty; -import java.io.File; import java.util.HashMap; import java.util.HashSet; import java.util.Map; @@ -73,10 +71,6 @@ public Map getBaseConfiguration(final String graphName, final Cl put(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_PROPERTY_ID_MANAGER, idMaker); if (requiresListCardinalityAsDefault(loadGraphWith, test, testMethodName)) put(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.list.name()); - if (requiresPersistence(test, testMethodName)) { - put(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - put(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION,TestHelper.makeTestDataFile(test, "temp", testMethodName + ".kryo")); - } }}; } @@ -84,13 +78,6 @@ public Map getBaseConfiguration(final String graphName, final Cl public void clear(final Graph graph, final Configuration configuration) throws Exception { if (graph != null) graph.close(); - - // in the even the graph is persisted we need to clean up - final String graphLocation = null != configuration ? configuration.getString(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null) : null; - if (graphLocation != null) { - final File f = new File(graphLocation); - f.delete(); - } } @Override @@ -98,13 +85,6 @@ public Set getImplementations() { return IMPLEMENTATION; } - /** - * Determines if a test requires TinkerGraph persistence to be configured with graph location and format. - */ - protected static boolean requiresPersistence(final Class test, final String testMethodName) { - return test == GraphTest.class && testMethodName.equals("shouldPersistDataOnClose"); - } - /** * Determines if a test requires a different cardinality as the default or not. */ diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphUUIDProvider.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphUUIDProvider.java index 8a127e8330c..ef37147838f 100644 --- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphUUIDProvider.java +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphUUIDProvider.java @@ -20,13 +20,11 @@ package org.apache.tinkerpop.gremlin.tinkergraph; import org.apache.tinkerpop.gremlin.LoadGraphWith; -import org.apache.tinkerpop.gremlin.TestHelper; import org.apache.tinkerpop.gremlin.structure.Graph; import org.apache.tinkerpop.gremlin.structure.VertexProperty; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerMemoryGraph; -import java.io.File; import java.util.HashMap; import java.util.Map; @@ -47,10 +45,6 @@ public Map getBaseConfiguration(final String graphName, final Cl put(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_PROPERTY_ID_MANAGER, idMaker); if (requiresListCardinalityAsDefault(loadGraphWith, test, testMethodName)) put(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.list.name()); - if (requiresPersistence(test, testMethodName)) { - put(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - put(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, TestHelper.makeTestDataFile(test, "temp", testMethodName + ".kryo")); - } }}; } } diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerStorageGraphProvider.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerStorageGraphProvider.java index 6f777a630c2..d7f75e63442 100644 --- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerStorageGraphProvider.java +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerStorageGraphProvider.java @@ -70,8 +70,9 @@ public Map getBaseConfiguration(final String graphName, final Cl if (requiresListCardinalityAsDefault(loadGraphWith, test, testMethodName)) put(TinkerStorageGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.list.name()); if (requiresPersistence(test, testMethodName)) { - put(TinkerStorageGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - put(TinkerStorageGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION,TestHelper.makeTestDataFile(test, "temp", testMethodName + ".kryo")); + put(TinkerStorageGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + put(TinkerStorageGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, + TestHelper.makeTestDataDirectory(test, "temp", testMethodName)); } }}; } @@ -81,14 +82,26 @@ public void clear(final Graph graph, final Configuration configuration) throws E if (graph != null) graph.close(); - // in the even the graph is persisted we need to clean up - final String graphLocation = null != configuration ? configuration.getString(TinkerStorageGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null) : null; - if (graphLocation != null) { - final File f = new File(graphLocation); - f.delete(); + // in the event the graph is persisted we need to clean up the storage directory + final String storageDirectory = null != configuration ? configuration.getString(TinkerStorageGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, null) : null; + if (storageDirectory != null) { + deleteRecursively(new File(storageDirectory)); } } + private static void deleteRecursively(final File file) { + if (!file.exists()) + return; + if (file.isDirectory()) { + final File[] children = file.listFiles(); + if (children != null) { + for (final File child : children) + deleteRecursively(child); + } + } + file.delete(); + } + @Override public Set getImplementations() { return IMPLEMENTATION; diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraphTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraphTest.java index 536f5b5ee81..95856f60202 100644 --- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraphTest.java +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraphTest.java @@ -21,7 +21,6 @@ import org.apache.commons.configuration2.BaseConfiguration; import org.apache.commons.configuration2.Configuration; import org.apache.tinkerpop.gremlin.GraphHelper; -import org.apache.tinkerpop.gremlin.TestHelper; import org.apache.tinkerpop.gremlin.process.computer.Computer; import org.apache.tinkerpop.gremlin.process.traversal.P; import org.apache.tinkerpop.gremlin.process.traversal.Traversal; @@ -39,7 +38,6 @@ import org.apache.tinkerpop.gremlin.structure.T; import org.apache.tinkerpop.gremlin.structure.Vertex; import org.apache.tinkerpop.gremlin.structure.VertexProperty; -import org.apache.tinkerpop.gremlin.structure.io.Io; import org.apache.tinkerpop.gremlin.structure.io.GraphReader; import org.apache.tinkerpop.gremlin.structure.io.GraphWriter; import org.apache.tinkerpop.gremlin.structure.io.IoCore; @@ -63,11 +61,8 @@ import org.junit.Test; import java.awt.Color; -import java.io.BufferedOutputStream; import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; -import java.io.File; -import java.io.FileOutputStream; import java.io.InputStream; import java.util.ArrayList; import java.util.Arrays; @@ -80,7 +75,6 @@ import java.util.Set; import java.util.UUID; import java.util.concurrent.TimeUnit; -import java.util.function.Consumer; import java.util.function.Supplier; import static org.apache.tinkerpop.gremlin.process.traversal.AnonymousTraversalSource.traversal; @@ -95,7 +89,6 @@ import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; import static org.junit.Assume.assumeThat; -import static org.mockito.Mockito.mock; /** * @author Marko A. Rodriguez (http://markorodriguez.com) @@ -390,13 +383,6 @@ public void shouldSerializeTinkerGraphToGraphSONWithTypes() throws Exception { } } - @Test(expected = IllegalStateException.class) - public void shouldRequireGraphLocationIfFormatIsSet() { - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "graphml"); - TinkerGraph.open(conf); - } - @Test(expected = IllegalStateException.class) public void shouldNotModifyAVertexThatWasRemoved() { final TinkerGraph graph = TinkerGraph.open(); @@ -431,137 +417,6 @@ public void shouldNotReadValueOfPropertyOnVertexThatWasRemoved() { v.value("name"); } - @Test(expected = IllegalStateException.class) - public void shouldRequireGraphFormatIfLocationIsSet() { - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, TestHelper.makeTestDataDirectory(TinkerMemoryGraphTest.class)); - TinkerGraph.open(conf); - } - - @Test - public void shouldPersistToGraphML() { - final String graphLocation = TestHelper.makeTestDataFile(TinkerMemoryGraphTest.class, "shouldPersistToGraphML.xml"); - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "graphml"); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateModern(graph); - graph.close(); - - final TinkerGraph reloadedGraph = TinkerGraph.open(conf); - IoTest.assertModernGraph(reloadedGraph, true, true); - reloadedGraph.close(); - } - - @Test - public void shouldPersistToGraphSON() { - final String graphLocation = TestHelper.makeTestDataFile(TinkerMemoryGraphTest.class, "shouldPersistToGraphSON.json"); - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "graphson"); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateModern(graph); - graph.close(); - - final TinkerGraph reloadedGraph = TinkerGraph.open(conf); - IoTest.assertModernGraph(reloadedGraph, true, false); - reloadedGraph.close(); - } - - @Test - public void shouldPersistToGryo() { - final String graphLocation = TestHelper.makeTestDataFile(TinkerMemoryGraphTest.class, "shouldPersistToGryo.kryo"); - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateModern(graph); - graph.close(); - - final TinkerGraph reloadedGraph = TinkerGraph.open(conf); - IoTest.assertModernGraph(reloadedGraph, true, false); - reloadedGraph.close(); - } - - @Test - public void shouldPersistToGryoAndHandleMultiProperties() { - final String graphLocation = TestHelper.makeTestDataFile(TinkerMemoryGraphTest.class, "shouldPersistToGryoMulti.kryo"); - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateTheCrew(graph); - graph.close(); - - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.list.toString()); - final TinkerGraph reloadedGraph = TinkerGraph.open(conf); - IoTest.assertCrewGraph(reloadedGraph, false); - reloadedGraph.close(); - } - - @Test - public void shouldPersistWithRelativePath() { - final String graphLocation = TestHelper.convertToRelative(TinkerMemoryGraphTest.class, - TestHelper.makeTestDataPath(TinkerMemoryGraphTest.class)) - + "shouldPersistToGryoRelative.kryo"; - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateModern(graph); - graph.close(); - - final TinkerGraph reloadedGraph = TinkerGraph.open(conf); - IoTest.assertModernGraph(reloadedGraph, true, false); - reloadedGraph.close(); - } - - @Test - public void shouldPersistToAnyGraphFormat() { - final String graphLocation = TestHelper.makeTestDataFile(TinkerMemoryGraphTest.class, "shouldPersistToAnyGraphFormat.dat"); - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, TestIoBuilder.class.getName()); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateModern(graph); - - //Test write graph - graph.close(); - assertEquals(TestIoBuilder.calledOnMapper, 1); - assertEquals(TestIoBuilder.calledGraph, 1); - assertEquals(TestIoBuilder.calledCreate, 1); - - try (BufferedOutputStream os = new BufferedOutputStream(new FileOutputStream(f))){ - os.write("dummy string".getBytes()); - } catch (Exception e) { - e.printStackTrace(); - } - - //Test read graph - final TinkerGraph readGraph = TinkerGraph.open(conf); - assertEquals(TestIoBuilder.calledOnMapper, 1); - assertEquals(TestIoBuilder.calledGraph, 1); - assertEquals(TestIoBuilder.calledCreate, 1); - } - @Test public void shouldSerializeWithColorClassResolverToTinkerGraph() throws Exception { final Map colors = new HashMap<>(); @@ -1060,41 +915,6 @@ public Registration getRegistration(final Class clazz) { } } - public static class TestIoBuilder implements Io.Builder { - - static int calledGraph, calledCreate, calledOnMapper; - - public TestIoBuilder(){ - //Looks awkward to reset static vars inside a constructor, but makes sense from testing perspective - calledGraph = 0; - calledCreate = 0; - calledOnMapper = 0; - } - - @Override - public Io.Builder onMapper(final Consumer onMapper) { - calledOnMapper++; - return this; - } - - @Override - public Io.Builder graph(final Graph graph) { - calledGraph++; - return this; - } - - @Override - public Io create() { - calledCreate++; - return mock(Io.class); - } - - @Override - public boolean requiresVersion(final Object version) { - return false; - } - } - @Test public void shouldGroupMultiPropertyValuesUnderSingleKeyForPropertyMap() throws Exception { try (final TinkerGraph graph = TinkerGraph.open()) { diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraphTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraphTest.java index af5e79124a1..67bb6bffdd6 100644 --- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraphTest.java +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraphTest.java @@ -1415,14 +1415,16 @@ public void shouldHandleConcurrentChangeForVertexMetaProperty() throws Interrupt try { gtx.tx().commit(); - //fail("should throw TransactionException"); + fail("should throw TransactionException"); } catch (TransactionException e) { } - // should be only 1 vertex with updated property + // tx2 committed first and wins; tx1 conflicted and rolled back, so its meta1 was never persisted assertEquals(1L, (long) gtx.V().count().next()); - assertEquals("tx2", gtx.V(v1.id()).properties("test").values("meta1", "meta2").next()); + assertEquals(1L, (long) gtx.V(v1.id()).properties("test").properties().count().next()); + assertEquals("tx2", gtx.V(v1.id()).properties("test").values("meta2").next()); + assertFalse(gtx.V(v1.id()).properties("test").values("meta1").hasNext()); } @Test diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java new file mode 100644 index 00000000000..9a304f4428a --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java @@ -0,0 +1,646 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.Edge; +import org.apache.tinkerpop.gremlin.structure.Graph; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.structure.VertexProperty; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.math.BigDecimal; +import java.math.BigInteger; +import java.time.Duration; +import java.time.OffsetDateTime; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +/** + * Engine-agnostic conformance suite ("TCK") for pluggable {@link TinkerStorage} engines. A concrete engine is tested + * by subclassing this and returning the {@code gremlin.tinkergraph.storage} value that selects it (an engine name or a + * fully-qualified class name). Every test opens a {@link TinkerStorageGraph} backed by a fresh temporary directory, + * mutates it, reopens from the same configuration, and asserts the data survived. A new engine drops in by adding one + * subclass. + */ +public abstract class AbstractTinkerStorageConformanceTest { + + @Rule + public TemporaryFolder tempFolder = new TemporaryFolder(); + + private String location; + + /** + * The {@code gremlin.tinkergraph.storage} configuration value that selects the engine under test. + */ + protected abstract String storageEngine(); + + @Before + public void setUp() throws Exception { + location = tempFolder.newFolder("storage").getAbsolutePath(); + } + + protected Configuration config() { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(Graph.GRAPH, TinkerStorageGraph.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, storageEngine()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, location); + return conf; + } + + protected TinkerStorageGraph open() { + return TinkerStorageGraph.open(config()); + } + + @Test + public void shouldPersistVerticesAndEdgesAcrossReopen() { + TinkerStorageGraph graph = open(); + final Vertex marko = graph.addVertex(T.id, 1, T.label, "person", "name", "marko", "age", 29); + final Vertex lop = graph.addVertex(T.id, 3, T.label, "software", "name", "lop", "lang", "java"); + marko.addEdge("created", lop, T.id, 9, "weight", 0.4); + graph.tx().commit(); + graph.close(); + + graph = open(); + assertEquals(2, countOf(graph.vertices())); + assertEquals(1, countOf(graph.edges())); + final Vertex reMarko = graph.vertices(1).next(); + assertEquals("marko", reMarko.value("name")); + assertEquals(Integer.valueOf(29), reMarko.value("age")); + final Edge reCreated = graph.edges(9).next(); + assertEquals("created", reCreated.label()); + assertEquals(0.4, reCreated.value("weight"), 0.0001); + assertEquals(Integer.valueOf(1), reCreated.outVertex().id()); + assertEquals(Integer.valueOf(3), reCreated.inVertex().id()); + graph.close(); + } + + @Test + public void shouldPersistAcrossMultipleCommits() { + TinkerStorageGraph graph = open(); + for (int i = 0; i < 10; i++) { + graph.addVertex(T.id, i, "value", i); + graph.tx().commit(); + } + graph.close(); + + graph = open(); + assertEquals(10, countOf(graph.vertices())); + for (int i = 0; i < 10; i++) + assertEquals(Integer.valueOf(i), graph.vertices(i).next().value("value")); + graph.close(); + } + + @Test + public void shouldPersistModificationsWithLastWriteWinning() { + TinkerStorageGraph graph = open(); + final Vertex v = graph.addVertex(T.id, 1, "name", "original"); + graph.tx().commit(); + v.property("name", "updated"); + graph.tx().commit(); + graph.close(); + + graph = open(); + assertEquals("updated", graph.vertices(1).next().value("name")); + graph.close(); + } + + @Test + public void shouldNotPersistRemovedElements() { + TinkerStorageGraph graph = open(); + final Vertex a = graph.addVertex(T.id, 1); + final Vertex b = graph.addVertex(T.id, 2); + final Edge e = a.addEdge("knows", b, T.id, 10); + graph.tx().commit(); + e.remove(); + b.remove(); + graph.tx().commit(); + graph.close(); + + graph = open(); + assertEquals(1, countOf(graph.vertices())); + assertEquals(0, countOf(graph.edges())); + assertNotNull(graph.vertices(1).next()); + assertFalse(graph.vertices(2).hasNext()); + graph.close(); + } + + @Test + public void shouldNotPersistRolledBackTransaction() { + TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1, "name", "committed"); + graph.tx().commit(); + graph.addVertex(T.id, 2, "name", "rolledback"); + graph.tx().rollback(); + graph.close(); + + graph = open(); + assertEquals(1, countOf(graph.vertices())); + assertNotNull(graph.vertices(1).next()); + assertFalse(graph.vertices(2).hasNext()); + graph.close(); + } + + @Test + public void shouldPersistMetaPropertiesAndMultiProperties() { + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.list.name()); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + final Vertex v = graph.addVertex(T.id, 1); + final VertexProperty vp = v.property(VertexProperty.Cardinality.list, "name", "marko"); + vp.property("acl", "public"); + v.property(VertexProperty.Cardinality.list, "name", "marko a. rodriguez"); + graph.tx().commit(); + graph.close(); + + graph = TinkerStorageGraph.open(conf); + final Vertex reV = graph.vertices(1).next(); + assertEquals(2, countOf(reV.properties("name"))); + final Iterator> props = reV.properties("name"); + boolean foundAcl = false; + while (props.hasNext()) { + final VertexProperty p = props.next(); + if (p.properties("acl").hasNext()) { + assertEquals("public", p.properties("acl").next().value()); + foundAcl = true; + } + } + assertTrue("meta-property should survive persistence", foundAcl); + graph.close(); + } + + @Test + public void shouldPreserveStateAfterCompact() { + TinkerStorageGraph graph = open(); + for (int i = 0; i < 5; i++) { + graph.addVertex(T.id, i, "value", i); + graph.tx().commit(); + } + graph.compact(); + // keep writing after compaction to exercise the truncated log + graph.addVertex(T.id, 100, "value", 100); + graph.tx().commit(); + graph.close(); + + graph = open(); + assertEquals(6, countOf(graph.vertices())); + assertEquals(Integer.valueOf(100), graph.vertices(100).next().value("value")); + assertEquals(Integer.valueOf(3), graph.vertices(3).next().value("value")); + graph.close(); + } + + @Test + public void shouldReopenEmptyGraph() { + TinkerStorageGraph graph = open(); + graph.close(); + + graph = open(); + assertEquals(0, countOf(graph.vertices())); + assertEquals(0, countOf(graph.edges())); + graph.close(); + } + + @Test + public void shouldReportPersistenceFeature() { + final TinkerStorageGraph graph = open(); + assertTrue(graph.features().graph().supportsPersistence()); + graph.close(); + } + + @Test + public void shouldPersistConcurrentCommitsWithoutLossAcrossReopen() throws Exception { + // End-to-end companion to StorageCommitSerializationTest: many threads commit disjoint vertices at once and, + // on reopen, every record must survive. This exercises the real engine but cannot by itself *prove* the lock + // works — log corruption from interleaving is scheduling-dependent — so the deterministic guarantee is + // asserted separately by StorageCommitSerializationTest via a probe engine. + final int threads = 8; + final int commitsPerThread = 50; + final TinkerStorageGraph writeGraph = open(); + try { + final ExecutorService pool = Executors.newFixedThreadPool(threads); + final CountDownLatch start = new CountDownLatch(1); + final List> futures = new ArrayList<>(); + for (int t = 0; t < threads; t++) { + final int threadId = t; + futures.add(pool.submit(() -> { + start.await(); // release all threads together to maximize contention on the commit path + for (int i = 0; i < commitsPerThread; i++) { + final int id = threadId * commitsPerThread + i; + writeGraph.addVertex(T.id, id, "value", id); + writeGraph.tx().commit(); + } + return null; + })); + } + start.countDown(); + for (final Future f : futures) + f.get(60, TimeUnit.SECONDS); + pool.shutdown(); + assertTrue(pool.awaitTermination(60, TimeUnit.SECONDS)); + } finally { + writeGraph.close(); + } + + // reopen from disk: a corrupt (interleaved) log frame would throw or drop records here + final TinkerStorageGraph reopened = open(); + try { + final int expected = threads * commitsPerThread; + assertEquals(expected, countOf(reopened.vertices())); + for (int id = 0; id < expected; id++) + assertEquals(Integer.valueOf(id), reopened.vertices(id).next().value("value")); + } finally { + reopened.close(); + } + } + + @Test + public void shouldRoundTripDiverseValueTypes() { + final Map values = new LinkedHashMap<>(); + values.put("int", 42); + values.put("long", 42L); + values.put("float", 1.5f); + values.put("double", 2.5d); + values.put("bool", true); + values.put("byte", (byte) 7); + values.put("short", (short) 9); + values.put("char", 'x'); + values.put("string", "hello"); + values.put("uuid", new UUID(12L, 34L)); + values.put("bigint", new BigInteger("123456789012345678901234567890")); + values.put("bigdec", new BigDecimal("3.14159265358979")); + values.put("datetime", OffsetDateTime.parse("2020-01-02T03:04:05Z")); + values.put("duration", Duration.ofSeconds(90)); + + TinkerStorageGraph graph = open(); + try { + final Vertex v = graph.addVertex(T.id, 1); + values.forEach(v::property); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + final Vertex v = graph.vertices(1).next(); + values.forEach((k, expected) -> assertEquals(k, expected, v.value(k))); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripCollectionValuedProperties() { + final List list = Arrays.asList(1, "two", 3.0d); + final Map map = new LinkedHashMap<>(); + map.put("a", 1); + map.put("b", "two"); + final Set set = new LinkedHashSet<>(Arrays.asList("x", "y", "z")); + + TinkerStorageGraph graph = open(); + try { + final Vertex v = graph.addVertex(T.id, 1); + v.property("list", list); + v.property("map", map); + v.property("set", set); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + final Vertex v = graph.vertices(1).next(); + assertEquals(list, v.value("list")); + assertEquals(map, v.value("map")); + assertEquals(set, v.value("set")); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripNullPropertyValue() { + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES, true); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + try { + final Vertex v = graph.addVertex(T.id, 1); + v.property("maybe", null); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = TinkerStorageGraph.open(conf); + try { + final VertexProperty vp = graph.vertices(1).next().properties("maybe").next(); + assertTrue(vp.isPresent()); + assertNull(vp.value()); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripHeterogeneousSameKeyTypes() { + TinkerStorageGraph graph = open(); + try { + graph.addVertex(T.id, 1, "k", 42); // Integer + graph.addVertex(T.id, 2, "k", 42L); // Long + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + assertEquals(Integer.valueOf(42), graph.vertices(1).next().value("k")); + assertEquals(Long.valueOf(42L), graph.vertices(2).next().value("k")); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripUuidElementIds() { + roundTripElementIds(new UUID(0L, 1L), new UUID(0L, 2L), new UUID(0L, 10L)); + } + + @Test + public void shouldRoundTripStringElementIds() { + roundTripElementIds("v-1", "v-2", "e-10"); + } + + // uses the default ANY id manager so any id type is accepted verbatim; the point is that the storage codec + // round-trips non-Long element ids through its scalar id encoding + private void roundTripElementIds(final Object outId, final Object inId, final Object edgeId) { + TinkerStorageGraph graph = open(); + try { + final Vertex a = graph.addVertex(T.id, outId, "name", "a"); + final Vertex b = graph.addVertex(T.id, inId, "name", "b"); + a.addEdge("knows", b, T.id, edgeId, "weight", 0.5d); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + assertEquals("a", graph.vertices(outId).next().value("name")); + assertEquals(outId, graph.vertices(outId).next().id()); + final Edge e = graph.edges(edgeId).next(); + assertEquals("knows", e.label()); + assertEquals(outId, e.outVertex().id()); + assertEquals(inId, e.inVertex().id()); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripLargeSchemaAcrossVarintBoundary() { + // >127 distinct keys and >127 values under one key push dictionary refs and counts past the single-byte + // LEB128 range, exercising the multi-byte varint path that small graphs never reach + final int n = 200; + // list cardinality so the >127 values under "multi" survive reopen (reconstruction takes cardinality from + // graph config, not the stored record) + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, "list"); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + try { + final Vertex v = graph.addVertex(T.id, 1); + for (int i = 0; i < n; i++) + v.property("key" + i, i); + for (int i = 0; i < n; i++) + v.property(VertexProperty.Cardinality.list, "multi", i); + graph.tx().commit(); + graph.compact(); // also exercises a dictionary header with >127 entries + } finally { + graph.close(); + } + graph = TinkerStorageGraph.open(conf); + try { + final Vertex v = graph.vertices(1).next(); + for (int i = 0; i < n; i++) + assertEquals("key" + i, Integer.valueOf(i), v.value("key" + i)); + assertEquals(n, countOf(v.properties("multi"))); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripUnicodeKeysAndValues() { + TinkerStorageGraph graph = open(); + try { + final Vertex v = graph.addVertex(T.id, 1); + v.property("naïve", "café"); + v.property("日本語", "テスト"); + v.property("emoji", "party🎉"); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + final Vertex v = graph.vertices(1).next(); + assertEquals("café", v.value("naïve")); + assertEquals("テスト", v.value("日本語")); + assertEquals("party🎉", v.value("emoji")); + } finally { + graph.close(); + } + } + + @Test + public void shouldSurviveRepeatedCompactionCycles() { + TinkerStorageGraph graph = open(); + try { + graph.addVertex(T.id, 1, "a", 1); + graph.tx().commit(); + graph.compact(); + graph.addVertex(T.id, 2, "b", 2); + graph.tx().commit(); + graph.compact(); + graph.addVertex(T.id, 3, "c", 3); + graph.tx().commit(); + graph.compact(); + } finally { + graph.close(); + } + graph = open(); + try { + assertEquals(3, countOf(graph.vertices())); + assertEquals(Integer.valueOf(1), graph.vertices(1).next().value("a")); + assertEquals(Integer.valueOf(2), graph.vertices(2).next().value("b")); + assertEquals(Integer.valueOf(3), graph.vertices(3).next().value("c")); + } finally { + graph.close(); + } + } + + @Test + public void shouldSurviveMultipleOpenCloseSessions() { + TinkerStorageGraph graph = open(); + try { + graph.addVertex(T.id, 1, "n", "one"); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + graph.addVertex(T.id, 2, "n", "two"); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + graph.addVertex(T.id, 3, "n", "three"); + graph.tx().commit(); + graph.compact(); + } finally { + graph.close(); + } + graph = open(); + try { + assertEquals(3, countOf(graph.vertices())); + assertEquals("one", graph.vertices(1).next().value("n")); + assertEquals("two", graph.vertices(2).next().value("n")); + assertEquals("three", graph.vertices(3).next().value("n")); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripConcurrentCommitsWithDistinctKeys() throws Exception { + // concurrent commits that each introduce a distinct property key stress dictionary growth under the + // commit-write lock; on reopen every distinct key must resolve + final int threads = 8; + final int perThread = 25; + final TinkerStorageGraph writeGraph = open(); + try { + final ExecutorService pool = Executors.newFixedThreadPool(threads); + final CountDownLatch start = new CountDownLatch(1); + final List> futures = new ArrayList<>(); + for (int t = 0; t < threads; t++) { + final int threadId = t; + futures.add(pool.submit(() -> { + start.await(); + for (int i = 0; i < perThread; i++) { + final int id = threadId * perThread + i; + writeGraph.addVertex(T.id, id, "k_" + threadId + "_" + i, id); + writeGraph.tx().commit(); + } + return null; + })); + } + start.countDown(); + for (final Future f : futures) + f.get(60, TimeUnit.SECONDS); + pool.shutdown(); + assertTrue(pool.awaitTermination(60, TimeUnit.SECONDS)); + } finally { + writeGraph.close(); + } + final TinkerStorageGraph reopened = open(); + try { + final int expected = threads * perThread; + assertEquals(expected, countOf(reopened.vertices())); + for (int t = 0; t < threads; t++) { + for (int i = 0; i < perThread; i++) { + final int id = threadId(t, i, perThread); + assertEquals(Integer.valueOf(id), reopened.vertices(id).next().value("k_" + t + "_" + i)); + } + } + } finally { + reopened.close(); + } + } + + private static int threadId(final int t, final int i, final int perThread) { + return t * perThread + i; + } + + private static long countOf(final Iterator it) { + long count = 0; + while (it.hasNext()) { + it.next(); + count++; + } + return count; + } + + @Test + public void shouldNotPersistUncommittedDataOnClose() { + // close() compacts, and compaction must snapshot only committed state. Uncommitted mutations left on the + // closing thread must not reach disk, matching how a crash drops them. + TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1, "name", "marko"); + graph.addVertex(T.id, 2, "name", "vadas"); + // deliberately no commit + graph.close(); + + graph = open(); + assertEquals("uncommitted vertices must not survive a graceful close", 0, countOf(graph.vertices())); + graph.close(); + } + + @Test + public void shouldPersistOnlyCommittedPortionOnClose() { + // a committed vertex plus later uncommitted work: only the committed portion is durable across a graceful close. + TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1, "name", "marko"); + graph.tx().commit(); + graph.addVertex(T.id, 2, "name", "vadas"); // uncommitted + graph.vertices(1).next().property("age", 29); // uncommitted modification to a committed vertex + graph.close(); + + graph = open(); + assertEquals("only the committed vertex is durable", 1, countOf(graph.vertices())); + final Vertex marko = graph.vertices(1).next(); + assertEquals("marko", marko.value("name")); + assertFalse("the uncommitted property must not survive", marko.properties("age").hasNext()); + graph.close(); + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/ConcurrencyProbeStorage.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/ConcurrencyProbeStorage.java new file mode 100644 index 00000000000..21b6c014da2 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/ConcurrencyProbeStorage.java @@ -0,0 +1,82 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.AbstractTinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerEdge; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex; + +import java.util.Collection; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * A {@link TinkerStorage} test double that deterministically detects whether the commit path serializes writes. It + * persists nothing; instead {@link #persist} tracks how many threads are inside it at once and records a violation if + * ever more than one is. It also sleeps briefly while "inside" to widen the window, so an unserialized commit path + * trips the detector reliably rather than racily. Selected by fully-qualified class name via + * {@code gremlin.tinkergraph.storage}. State is static because the engine is instantiated reflectively. + */ +public final class ConcurrencyProbeStorage implements TinkerStorage { + + static final AtomicInteger inFlight = new AtomicInteger(0); + static final AtomicInteger maxObserved = new AtomicInteger(0); + static final AtomicInteger concurrentEntries = new AtomicInteger(0); + + static void reset() { + inFlight.set(0); + maxObserved.set(0); + concurrentEntries.set(0); + } + + @Override + public void open(final AbstractTinkerGraph graph, final Configuration config) { } + + @Override + public void replay(final AbstractTinkerGraph graph) { } + + @Override + public void persist(final long txVersion, + final Collection> changedVertices, + final Collection> changedEdges) { + final int concurrent = inFlight.incrementAndGet(); + try { + maxObserved.accumulateAndGet(concurrent, Math::max); + if (concurrent > 1) + concurrentEntries.incrementAndGet(); + // widen the window so an unserialized path is caught deterministically, not by luck + try { + Thread.sleep(1); + } catch (InterruptedException ie) { + Thread.currentThread().interrupt(); + } + } finally { + inFlight.decrementAndGet(); + } + } + + @Override + public void flush() { } + + @Override + public void compact(final AbstractTinkerGraph graph) { } + + @Override + public void close() { } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DirectoryLockTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DirectoryLockTest.java new file mode 100644 index 00000000000..02e2fc24c51 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DirectoryLockTest.java @@ -0,0 +1,101 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.Graph; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.File; +import java.nio.channels.FileChannel; +import java.nio.channels.FileLock; +import java.nio.file.StandardOpenOption; + +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * Verifies that a {@link TinkerStorageGraph} takes an exclusive lock on its storage directory so a second opener on + * the same location fails fast rather than corrupting the store. + */ +public class DirectoryLockTest { + + @Rule + public TemporaryFolder tempFolder = new TemporaryFolder(); + + private String location; + + @Before + public void setUp() throws Exception { + location = tempFolder.newFolder("storage").getAbsolutePath(); + } + + private Configuration config() { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(Graph.GRAPH, TinkerStorageGraph.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, location); + return conf; + } + + @Test + public void shouldReleaseLockOnCloseAndAllowReopen() { + // sequential open/close must not leave the directory wedged + TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.close(); + + graph = TinkerStorageGraph.open(config()); + assertNotNull(graph.vertices(1).next()); + graph.close(); + } + + @Test + public void shouldRejectSecondOpenWhileLocationIsLocked() throws Exception { + // simulate another process holding the directory lock by taking the OS lock on the LOCK file directly + final File dir = new File(location); + dir.mkdirs(); + final File lockFile = new File(dir, DirectoryLock.LOCK_FILE); + try (final FileChannel channel = FileChannel.open(lockFile.toPath(), + StandardOpenOption.CREATE, StandardOpenOption.WRITE); + final FileLock held = channel.lock()) { + assertNotNull(held); + try { + TinkerStorageGraph.open(config()); + fail("expected open to fail while the storage location is locked"); + } catch (IllegalStateException expected) { + assertTrue("message should name the location: " + expected.getMessage(), + expected.getMessage().contains(location)); + } + } + + // once the simulated holder releases (try-with-resources above), the location opens normally + final TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.close(); + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java new file mode 100644 index 00000000000..365784528d2 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java @@ -0,0 +1,822 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.Edge; +import org.apache.tinkerpop.gremlin.structure.Graph; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.structure.VertexProperty; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; +import org.junit.Test; + +import java.io.DataInputStream; +import java.io.File; +import java.io.IOException; +import java.io.RandomAccessFile; +import java.nio.ByteBuffer; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.util.Collections; +import java.util.HashMap; +import java.util.Iterator; +import java.util.Map; +import java.util.zip.CRC32; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * Runs the shared {@link AbstractTinkerStorageConformanceTest} suite against the {@link GraphBinaryStorage} engine and + * adds engine-specific tests for the on-disk log layout, sync modes, auto-compaction, snapshot streaming, and + * corruption detection. + *

+ * Not covered here (deliberately): true {@code fsync} durability against OS crash or power loss. The {@code commit} + * vs. {@code os} sync-mode tests verify configuration and a graceful round-trip, but a JVM unit test cannot prove that + * an acknowledged commit survives a kernel crash — that needs OS-level fault injection (e.g. a FUSE layer that drops + * un-synced writes, or {@code dm-flakey}), which is out of scope. Crash-*consistency* of the file layout (as opposed + * to device-level durability) is covered deterministically by {@link StorageCrashConsistencyTest}. + */ +public class GraphBinaryStorageTest extends AbstractTinkerStorageConformanceTest { + + @Override + protected String storageEngine() { + return "graphbinary"; + } + + @Test + public void shouldWriteSnapshotAndLogFiles() { + final TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1); + graph.tx().commit(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + assertTrue(new File(location, GraphBinaryStorage.LOG_FILE).exists()); + graph.close(); + // close compacts, producing a snapshot and truncating the log + assertTrue(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).exists()); + } + + @Test + public void shouldPersistWithOsSyncMode() { + // 'os' is a weaker durability mode (no fsync); a graceful close/reopen must still round-trip the data. The + // OS-crash-loss window that distinguishes it from 'commit' cannot be exercised in a unit test. + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_SYNC, "os"); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1, "value", 42); + graph.tx().commit(); + graph.close(); + + graph = TinkerStorageGraph.open(conf); + assertEquals(1, countOf(graph.vertices())); + assertEquals(Integer.valueOf(42), graph.vertices(1).next().value("value")); + graph.close(); + } + + @Test + public void shouldPersistWithDefaultCommitSyncMode() { + // with no sync mode configured the engine defaults to 'commit' (fsync per commit); data must round-trip. + TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1, "value", 42); + graph.tx().commit(); + graph.close(); + + graph = open(); + assertEquals(Integer.valueOf(42), graph.vertices(1).next().value("value")); + graph.close(); + } + + @Test(expected = IllegalArgumentException.class) + public void shouldRejectUnknownSyncMode() { + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_SYNC, "bogus"); + TinkerStorageGraph.open(conf); + } + + @Test + public void shouldLeaveNoTempSnapshotAfterCompaction() { + final TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1, "value", 1); + graph.tx().commit(); + graph.compact(); + // the atomic rename must consume the temp file, leaving a durable snapshot and no leftover .tmp + assertTrue(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).exists()); + assertTrue(!new File(location, GraphBinaryStorage.SNAPSHOT_FILE + ".tmp").exists()); + graph.close(); + } + + @Test + public void shouldStreamSnapshotAsOneFramePerElement() throws Exception { + final TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + final Vertex a = graph.addVertex(T.id, 1, "name", "a"); + final Vertex b = graph.addVertex(T.id, 2, "name", "b"); + final Vertex c = graph.addVertex(T.id, 3, "name", "c"); + a.addEdge("knows", b, T.id, 10); + b.addEdge("knows", c, T.id, 11); + graph.tx().commit(); + graph.compact(); + + // the snapshot must be streamed: a dictionary header frame plus one framed record per element (3 vertices + + // 2 edges = 5), rather than a single whole-graph frame, so compaction never buffers the entire graph at once + final File snapshotFile = new File(location, GraphBinaryStorage.SNAPSHOT_FILE); + assertEquals(1 + 5, countFrames(snapshotFile)); + graph.close(); + + // and the streamed snapshot must reopen to exactly the same graph + final TinkerStorageGraph reopened = open(); + assertEquals(3, countOf(reopened.vertices())); + assertEquals(2, countOf(reopened.edges())); + assertEquals("a", reopened.vertices(1).next().value("name")); + assertEquals("knows", reopened.edges(10).next().label()); + reopened.close(); + } + + @Test + public void shouldStreamSnapshotFrameByFrameAtScale() { + // Bounded-memory proxy for the streaming snapshot path: rather than measure heap (flaky, JVM-dependent), assert + // the observable streaming property holds at scale — a large graph is written as a dictionary header frame plus + // one frame per element, never one whole-graph frame — and round-trips intact. This is the property that keeps + // compaction from materializing a second full copy of the graph in memory; it is not a hard OOM assertion. + final int vertexCount = 500; + final int edgeCount = 499; + final TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + try { + for (int i = 0; i < vertexCount; i++) + graph.addVertex(T.id, i, "value", i); + for (int i = 0; i < edgeCount; i++) + graph.vertices(i).next().addEdge("next", graph.vertices(i + 1).next(), T.id, 1_000_000 + i); + graph.tx().commit(); + graph.compact(); + + try { + assertEquals(1 + vertexCount + edgeCount, countFrames(new File(location, GraphBinaryStorage.SNAPSHOT_FILE))); + } catch (Exception ex) { + throw new RuntimeException(ex); + } + } finally { + graph.close(); + } + + final TinkerStorageGraph reopened = open(); + try { + assertEquals(vertexCount, countOf(reopened.vertices())); + assertEquals(edgeCount, countOf(reopened.edges())); + assertEquals(Integer.valueOf(499), reopened.vertices(499).next().value("value")); + } finally { + reopened.close(); + } + } + + /** + * Count the framed records in a storage file: a fixed header ({@code HEADER_SIZE} bytes) followed by frames of a + * 4-byte big-endian payload length, a 4-byte CRC, then that many payload bytes. + */ + private static int countFrames(final File file) throws Exception { + int frames = 0; + try (final DataInputStream in = new DataInputStream(new java.io.BufferedInputStream(new java.io.FileInputStream(file)))) { + if (!skipFully(in, GraphBinaryStorage.HEADER_SIZE)) return 0; + while (true) { + final int len; + try { + len = in.readInt(); + } catch (java.io.EOFException eof) { + break; + } + in.readInt(); // CRC + if (!skipFully(in, len)) break; + frames++; + } + } + return frames; + } + + /** + * Skip exactly {@code n} bytes, reading in a loop because a single {@link DataInputStream#skip} may skip fewer. + * Returns false if EOF is reached first. + */ + private static boolean skipFully(final DataInputStream in, final long n) throws Exception { + long left = n; + while (left > 0) { + final long s = in.skip(left); + if (s <= 0) { + if (in.read() < 0) return false; // genuine EOF + left -= 1; + } else { + left -= s; + } + } + return true; + } + + @Test + public void shouldAutoCompactWhenLogExceedsThreshold() { + // a small threshold makes automatic compaction fire mid-run, without any explicit compact()/close() + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, 2048L); + final String location = conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + try { + for (int i = 0; i < 200; i++) { + graph.addVertex(T.id, i, "value", i, "pad", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"); + graph.tx().commit(); + } + // auto-compaction should have folded the log into a snapshot and truncated it well below the total + // bytes written, so the live log stays bounded rather than growing with every commit + final File logFile = new File(location, GraphBinaryStorage.LOG_FILE); + final File snapshotFile = new File(location, GraphBinaryStorage.SNAPSHOT_FILE); + assertTrue("expected a snapshot from auto-compaction", snapshotFile.exists()); + assertTrue("expected the live log to stay bounded, was " + logFile.length(), + logFile.length() < 8192); + } finally { + graph.close(); + } + + // data must survive across reopen despite the mid-run compactions + graph = TinkerStorageGraph.open(conf); + try { + assertEquals(200, countOf(graph.vertices())); + assertEquals(Integer.valueOf(199), graph.vertices(199).next().value("value")); + } finally { + graph.close(); + } + } + + @Test + public void shouldNotAutoCompactWhenThresholdIsZero() { + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, 0L); + final String location = conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + try { + for (int i = 0; i < 50; i++) { + graph.addVertex(T.id, i, "pad", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"); + graph.tx().commit(); + } + // with auto-compaction disabled, no snapshot appears until close()/compact() + assertTrue(!new File(location, GraphBinaryStorage.SNAPSHOT_FILE).exists()); + } finally { + graph.close(); + } + } + + @Test + public void shouldRecoverFromTruncatedTrailingFrame() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1, "value", 1); + graph.tx().commit(); + graph.addVertex(T.id, 2, "value", 2); + graph.tx().commit(); + graph.tx().close(); + + // capture the raw log (two good frames) while the graph holds it, then close to release the directory lock. + // close() compacts, folding the log into a snapshot, so we reconstruct a "crashed" on-disk state below. + final File logFile = new File(location, GraphBinaryStorage.LOG_FILE); + final byte[] goodLog = Files.readAllBytes(logFile.toPath()); + graph.close(); + + // recreate the pre-crash layout: no snapshot, a log of the two good frames plus a torn trailing frame (a + // length prefix promising 100 bytes with only a couple following), as an interrupted append would leave. + Files.deleteIfExists(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).toPath()); + try (final RandomAccessFile raf = new RandomAccessFile(logFile, "rw")) { + raf.setLength(0); + raf.write(goodLog); + raf.writeInt(100); + raf.write(new byte[]{ 0x01, 0x02 }); + } + + // reopening must recover the two fully-committed vertices and ignore the torn frame + graph = open(); + assertEquals(2, countOf(graph.vertices())); + assertEquals(Integer.valueOf(1), graph.vertices(1).next().value("value")); + assertEquals(Integer.valueOf(2), graph.vertices(2).next().value("value")); + graph.close(); + } + + @Test + public void shouldRestoreIndexDefinitionsOnReopen() throws Exception { + TinkerStorageGraph graph = open(); + graph.createIndex("name", Vertex.class); + graph.createIndex("weight", Edge.class); + final Vertex marko = graph.addVertex(T.id, 1, "name", "marko"); + final Vertex josh = graph.addVertex(T.id, 2, "name", "josh"); + marko.addEdge("knows", josh, T.id, 10, "weight", 0.5d); + graph.tx().commit(); + graph.close(); + + graph = open(); + try { + // the definitions come back, and the index covers data written before the restart rather than only + // writes that follow it + assertEquals(Collections.singleton("name"), graph.getIndexedKeys(Vertex.class)); + assertEquals(Collections.singleton("weight"), graph.getIndexedKeys(Edge.class)); + assertEquals(1L, (long) graph.traversal().V().has("name", "josh").count().next()); + assertEquals(1L, (long) graph.traversal().E().has("weight", 0.5d).count().next()); + } finally { + graph.close(); + } + } + + @Test + public void shouldNotRestoreADroppedIndex() throws Exception { + TinkerStorageGraph graph = open(); + graph.createIndex("name", Vertex.class); + graph.addVertex(T.id, 1, "name", "marko"); + graph.tx().commit(); + graph.dropIndex("name", Vertex.class); + graph.close(); + + graph = open(); + try { + assertTrue("a dropped index must not come back", graph.getIndexedKeys(Vertex.class).isEmpty()); + // data is unaffected by the index being gone; the lookup just falls back to a scan + assertEquals(1L, (long) graph.traversal().V().has("name", "marko").count().next()); + } finally { + graph.close(); + } + } + + @Test + public void shouldOpenWithoutIndexesWhenDefinitionsAreUnreadable() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.createIndex("name", Vertex.class); + graph.addVertex(T.id, 1, "name", "marko"); + graph.tx().commit(); + graph.close(); + + // a corrupt sidecar must degrade to the behaviour before definitions were recorded at all, never block a + // store whose data is perfectly readable + Files.write(new File(location, IndexDefinitions.INDEX_FILE).toPath(), + "this is not an index definition".getBytes(StandardCharsets.UTF_8)); + + graph = open(); + try { + assertTrue(graph.getIndexedKeys(Vertex.class).isEmpty()); + assertEquals(1L, (long) graph.traversal().V().has("name", "marko").count().next()); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripIndexKeysNeedingEscapes() throws Exception { + final String awkward = "a\tb\nc\\d"; + TinkerStorageGraph graph = open(); + graph.createIndex(awkward, Vertex.class); + graph.addVertex(T.id, 1, awkward, "value"); + graph.tx().commit(); + graph.close(); + + graph = open(); + try { + assertEquals(Collections.singleton(awkward), graph.getIndexedKeys(Vertex.class)); + assertEquals(1L, (long) graph.traversal().V().has(awkward, "value").count().next()); + } finally { + graph.close(); + } + } + + @Test + public void shouldNotRecordIndexesWithoutAStorageEngine() { + // a TinkerStorageGraph with no engine is transactional but in-memory, so it must touch no disk at all + final Configuration conf = new BaseConfiguration(); + conf.setProperty(Graph.GRAPH, TinkerStorageGraph.class.getName()); + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + try { + graph.createIndex("name", Vertex.class); + graph.addVertex(T.id, 1, "name", "marko"); + graph.tx().commit(); + assertEquals(Collections.singleton("name"), graph.getIndexedKeys(Vertex.class)); + } finally { + graph.close(); + } + } + + @Test + public void shouldFailOnCorruptFrameWithBadCrc() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1, "value", 1); + graph.tx().commit(); + graph.addVertex(T.id, 2, "value", 2); + graph.tx().commit(); + graph.tx().close(); + + // preserve the raw log, then reconstruct it with a bit flipped inside the first frame's payload — a complete + // frame whose CRC no longer matches (distinct from a short trailing frame, which is tolerated as truncation) + final File logFile = new File(location, GraphBinaryStorage.LOG_FILE); + final byte[] log = Files.readAllBytes(logFile.toPath()); + graph.close(); + // header, then first frame's 4-byte length + 4-byte CRC, then payload — flip the first payload byte + final int firstPayloadByte = GraphBinaryStorage.HEADER_SIZE + 2 * Integer.BYTES; + log[firstPayloadByte] ^= 0x01; + Files.deleteIfExists(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).toPath()); + Files.write(logFile.toPath(), log); + + try { + open(); + fail("expected reopen to fail on a CRC mismatch"); + } catch (Exception expected) { + assertTrue("cause should report corruption: " + rootMessage(expected), + rootMessage(expected).contains("CRC mismatch")); + } + } + + @Test + public void shouldFailOnStringLongerThanItsFrame() throws Exception { + // a dictionary-append entry declaring a ~2GB string in a record holding no such bytes. The declared length + // must be checked before the array is allocated, otherwise this is an OutOfMemoryError rather than a + // reportable corrupt frame. + final byte[] payload = new byte[] { + 0x01, // entry count = 1 + 0x05, // OP_DICT_APPEND + 0x00, // dictionary id = 0 + (byte) 0xF0, (byte) 0xFF, (byte) 0xFF, (byte) 0xFF, 0x07 // varint length = 0x7FFFFFF0 + }; + + try { + openWithSyntheticFrame(payload); + fail("expected reopen to fail on a string longer than its frame"); + } catch (Exception expected) { + assertTrue("cause should report corruption: " + rootMessage(expected), + rootMessage(expected).contains("string of")); + } + } + + @Test + public void shouldFailOnOverLongVarInt() throws Exception { + // a varint whose continuation bits run past the width of an int. Java masks a shift count to five bits, so + // without a bound this wraps and yields an arbitrary, possibly negative, value instead of failing. + final byte[] payload = new byte[] { + (byte) 0x80, (byte) 0x80, (byte) 0x80, (byte) 0x80, + (byte) 0x80, (byte) 0x80, (byte) 0x80, 0x00 + }; + + try { + openWithSyntheticFrame(payload); + fail("expected reopen to fail on an over-long varint"); + } catch (Exception expected) { + assertTrue("cause should report corruption: " + rootMessage(expected), + rootMessage(expected).contains("over-long varint")); + } + } + + @Test + public void shouldFailOnDictionaryRefWithNoSuchEntry() throws Exception { + // a vertex record whose label names dictionary entry 5 when the dictionary is empty. The ref must be + // validated rather than dereferenced straight into the backing list. + final byte[] payload = new byte[] { + 0x01, // entry count = 1 + 0x01, // OP_PUT_VERTEX + 0x01, 0x00, 0x00, 0x00, 0x2A, // id: GraphBinary INT tag then 42 + 0x01, // label count = 1 + 0x05 // dictionary ref = 5, nothing defined + }; + + try { + openWithSyntheticFrame(payload); + fail("expected reopen to fail on an undefined dictionary ref"); + } catch (Exception expected) { + assertTrue("cause should report corruption: " + rootMessage(expected), + rootMessage(expected).contains("dictionary ref")); + } + } + + /** + * Replace the store's log with a single well-formed frame (correct length prefix and CRC) carrying {@code + * payload}, then reopen. The framing must be valid so that replay reaches the codec rather than stopping at the + * frame checks, which are covered separately. + */ + private void openWithSyntheticFrame(final byte[] payload) throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.tx().close(); + graph.close(); + + final CRC32 crc = new CRC32(); + crc.update(payload); + final ByteBuffer frame = ByteBuffer.allocate(GraphBinaryStorage.HEADER_SIZE + 2 * Integer.BYTES + payload.length); + frame.put(GraphBinaryStorage.MAGIC); + frame.putInt(payload.length); + frame.putInt((int) crc.getValue()); + frame.put(payload); + + Files.deleteIfExists(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).toPath()); + Files.write(new File(location, GraphBinaryStorage.LOG_FILE).toPath(), frame.array()); + + open().close(); + } + + @Test + public void shouldFailOnForeignFileWithBadMagic() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.tx().close(); + final File logFile = new File(location, GraphBinaryStorage.LOG_FILE); + final byte[] log = Files.readAllBytes(logFile.toPath()); + graph.close(); + + // corrupt the magic so the file no longer identifies as a TinkerGraph storage file + log[0] ^= 0xFF; + Files.deleteIfExists(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).toPath()); + Files.write(logFile.toPath(), log); + + try { + open(); + fail("expected reopen to fail on bad magic"); + } catch (Exception expected) { + assertTrue("cause should report a bad storage file: " + rootMessage(expected), + rootMessage(expected).contains("not a TinkerGraph storage file")); + } + } + + @Test + public void shouldFailOnStoreWithUnsupportedVersionMarker() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.close(); + + // rewrite the store-level version marker to name a future, unsupported format version + final File versionFile = new File(location, GraphBinaryStorage.VERSION_FILE); + final byte[] marker = Files.readAllBytes(versionFile.toPath()); + marker[marker.length - 1] = 99; // version byte follows the magic + Files.write(versionFile.toPath(), marker); + + try { + open(); + fail("expected reopen to fail on an unsupported store version"); + } catch (Exception expected) { + assertTrue("cause should report the unsupported version and migration path: " + rootMessage(expected), + rootMessage(expected).contains("Unsupported storage format version") + && rootMessage(expected).contains("g.io()")); + } + } + + @Test + public void shouldRegenerateVertexPropertyIdsByDefault() { + // default: vertex-property ids are not persisted; the property still round-trips (value + meta), the id is + // simply reassigned on load + TinkerStorageGraph graph = open(); + try { + final VertexProperty vp = graph.addVertex(T.id, 1).property(VertexProperty.Cardinality.list, "name", "marko"); + vp.property("since", 2010); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + final VertexProperty vp = graph.vertices(1).next().properties("name").next(); + assertEquals("marko", vp.value()); + assertEquals(Integer.valueOf(2010), vp.property("since").value()); + } finally { + graph.close(); + } + } + + @Test + public void shouldPreserveVertexPropertyIdsWhenConfigured() { + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS, true); + final Object vpId; + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + try { + final VertexProperty vp = graph.addVertex(T.id, 1).property(VertexProperty.Cardinality.list, "name", "marko"); + vpId = vp.id(); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = TinkerStorageGraph.open(conf); + try { + final VertexProperty vp = graph.vertices(1).next().properties("name").next(); + assertEquals("marko", vp.value()); + assertEquals("vertex-property id should be preserved across reopen", vpId, vp.id()); + } finally { + graph.close(); + } + } + + @Test + public void shouldReplayDictionaryGrowthAcrossLogCommits() throws Exception { + // each commit introduces a new property key, so the dictionary grows via OP_DICT_APPEND across successive log + // frames. Reopening from a log with no snapshot must rebuild the dictionary incrementally and resolve all refs. + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, 0L); // keep the log, no auto-compaction + final String location = conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + for (int i = 0; i < 10; i++) { + graph.addVertex(T.id, i, "key" + i, i); + graph.tx().commit(); + } + graph.tx().close(); + final byte[] log = Files.readAllBytes(new File(location, GraphBinaryStorage.LOG_FILE).toPath()); + graph.close(); + + // restore a log-only store (no snapshot) and reopen + Files.deleteIfExists(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).toPath()); + Files.write(new File(location, GraphBinaryStorage.LOG_FILE).toPath(), log); + graph = TinkerStorageGraph.open(conf); + try { + assertEquals(10, countOf(graph.vertices())); + for (int i = 0; i < 10; i++) + assertEquals(Integer.valueOf(i), graph.vertices(i).next().value("key" + i)); + } finally { + graph.close(); + } + } + + @Test + public void shouldRewriteDictionaryOnCompactionAndReopen() { + TinkerStorageGraph graph = open(); + try { + for (int i = 0; i < 10; i++) + graph.addVertex(T.id, i, "key" + i, i); + graph.tx().commit(); + graph.compact(); // writes a fresh full-dictionary snapshot header, then element frames + } finally { + graph.close(); + } + graph = open(); + try { + assertEquals(10, countOf(graph.vertices())); + for (int i = 0; i < 10; i++) + assertEquals(Integer.valueOf(i), graph.vertices(i).next().value("key" + i)); + } finally { + graph.close(); + } + } + + @Test + public void shouldNotGrowDictionaryAcrossReopenCycles() throws Exception { + // With committed data and its key vocabulary held constant, repeated reopen+compact cycles must not grow the + // store. The dictionary numbering is rebuilt on replay, so a write session after a reopen resumes it rather + // than re-appending every live key as a duplicate. Before that fix the snapshot grew by the vocabulary size + // on every cycle. + final Configuration conf = config(); + final String location = conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + final File snapshot = new File(location, GraphBinaryStorage.SNAPSHOT_FILE); + + // seed a single vertex whose keys (person/name/age) are the entire vocabulary, then fold to a snapshot + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1, T.label, "person", "name", "marko", "age", 29); + graph.tx().commit(); + graph.compact(); + graph.close(); + final long seededSize = snapshot.length(); + + // reopen and re-compact repeatedly without changing the data; the snapshot must stay byte-for-byte the same size + for (int cycle = 0; cycle < 6; cycle++) { + graph = TinkerStorageGraph.open(conf); + graph.compact(); + final long size = snapshot.length(); + graph.close(); + assertEquals("snapshot grew on reopen+compact cycle " + cycle + " with unchanged data (dictionary bloat)", + seededSize, size); + } + + // and the data is still intact + graph = TinkerStorageGraph.open(conf); + try { + assertEquals(1, countOf(graph.vertices())); + assertEquals("marko", graph.vertices(1).next().value("name")); + } finally { + graph.close(); + } + } + + @Test + public void shouldStoreFewBytesPerElement() { + // regression guard: the dictionary-encoded format must stay well under the ~168 bytes/element the old + // whole-object format cost for a comparable graph (3 vertex props, 2 edge props, E=V). + final int vertexCount = 200; + final TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + try { + for (int i = 0; i < vertexCount; i++) + graph.addVertex(T.id, i, "name", "v" + i, "age", i % 100, "score", i * 1.5d); + for (int i = 0; i < vertexCount; i++) + graph.vertices(i).next().addEdge("knows", graph.vertices((i + 1) % vertexCount).next(), + T.id, 1_000_000 + i, "weight", i * 0.5d, "count", i % 7); + graph.tx().commit(); + graph.compact(); + final long bytes = new File(location, GraphBinaryStorage.SNAPSHOT_FILE).length(); + final double perElement = (double) bytes / (2 * vertexCount); + assertTrue("expected well under 168 bytes/element (whole-object baseline), got " + perElement, perElement < 100.0); + } finally { + graph.close(); + } + } + + // --- decode-error paths ----------------------------------------------------------------------------------------- + // A frame that clears CRC framing can still be internally malformed (a bug in an older writer, or a targeted flip + // that happens to keep the checksum valid). The codec must reject each such frame with a clear IOException rather + // than a silent wrong answer or an opaque runtime crash. These drive decodeFrame directly with hand-built frames; + // values below 128 are single-byte varints, so the frames are written as literal bytes. + + private static final byte OP_DEL_VERTEX = 2; + private static final byte OP_DICT_APPEND = 5; + + /** Decode one frame payload against a fresh codec, returning nothing — the maps are throwaway. */ + private static void decode(final GraphBinaryStorage codec, final byte[] frame) throws IOException { + final Map vertices = new HashMap<>(); + final Map edges = new HashMap<>(); + codec.decodeFrame(frame, vertices, edges); + } + + @Test + public void shouldRejectFrameWithUnknownOpCode() { + // entryCount=1, op=99 (no such op) + try { + decode(new GraphBinaryStorage(), new byte[]{ 1, 99 }); + fail("expected an IOException for an unknown op code"); + } catch (final IOException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("Unknown storage op code")); + } + } + + @Test + public void shouldRejectDictionaryAppendGap() { + // entryCount=1, OP_DICT_APPEND, id=5 into an empty dictionary (expected next id is 0) -> gap + try { + decode(new GraphBinaryStorage(), new byte[]{ 1, OP_DICT_APPEND, 5, 1, (byte) 'x' }); + fail("expected an IOException for a dictionary append gap"); + } catch (final IOException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("dictionary append gap")); + } + } + + @Test + public void shouldRejectDictionaryRedefinition() { + // append id 0 = "a", then re-append id 0 = "b" on the same codec -> redefinition (a mismatch, not an + // idempotent re-append of the same string) + final GraphBinaryStorage codec = new GraphBinaryStorage(); + try { + decode(codec, new byte[]{ 1, OP_DICT_APPEND, 0, 1, (byte) 'a' }); + decode(codec, new byte[]{ 1, OP_DICT_APPEND, 0, 1, (byte) 'b' }); + fail("expected an IOException for a redefined dictionary id"); + } catch (final IOException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("redefined")); + } + } + + @Test + public void shouldRejectUnknownValueTypeCode() { + // OP_DEL_VERTEX reads a scalar id; feed it a value type code (0xFF) no serializer claims. Without a guard this + // is an opaque NullPointerException; the codec must surface it as corruption instead. + try { + decode(new GraphBinaryStorage(), new byte[]{ 1, OP_DEL_VERTEX, (byte) 0xFF }); + fail("expected an IOException for an unknown value type code"); + } catch (final IOException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("unknown value type code")); + } + } + + private static String rootMessage(final Throwable t) { + Throwable cur = t; + while (cur.getCause() != null && cur.getCause() != cur) + cur = cur.getCause(); + return String.valueOf(cur.getMessage()); + } + + private static long countOf(final Iterator it) { + long count = 0; + while (it.hasNext()) { + it.next(); + count++; + } + return count; + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCommitSerializationTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCommitSerializationTest.java new file mode 100644 index 00000000000..1749b0ea7ea --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCommitSerializationTest.java @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.Graph; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +/** + * Deterministically verifies that {@link TinkerStorageGraph} serializes the durable write of concurrent commits. + * Because TinkerGraph transactions lock only their own changed elements, commits touching disjoint elements run their + * commit paths concurrently; the storage engine's single append log would interleave (corrupt) without the + * commit-write lock. Rather than rely on a race actually corrupting the log, this drives commits through + * {@link ConcurrencyProbeStorage}, which records whether two threads are ever inside {@code persist()} at once. With + * the lock that count is exactly zero; without it the probe trips reliably. + */ +public class StorageCommitSerializationTest { + + @Rule + public TemporaryFolder tempFolder = new TemporaryFolder(); + + private String location; + + @Before + public void setUp() throws Exception { + location = tempFolder.newFolder("storage").getAbsolutePath(); + ConcurrencyProbeStorage.reset(); + } + + private TinkerStorageGraph open() { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(Graph.GRAPH, TinkerStorageGraph.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, ConcurrencyProbeStorage.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, location); + return TinkerStorageGraph.open(conf); + } + + @Test + public void shouldSerializeConcurrentCommitWrites() throws Exception { + final int threads = 8; + final int commitsPerThread = 20; + final TinkerStorageGraph graph = open(); + try { + final ExecutorService pool = Executors.newFixedThreadPool(threads); + final CountDownLatch start = new CountDownLatch(1); + final List> futures = new ArrayList<>(); + for (int t = 0; t < threads; t++) { + final int threadId = t; + futures.add(pool.submit(() -> { + start.await(); + for (int i = 0; i < commitsPerThread; i++) { + graph.addVertex(T.id, threadId * commitsPerThread + i); + graph.tx().commit(); + } + return null; + })); + } + start.countDown(); + for (final Future f : futures) + f.get(60, TimeUnit.SECONDS); + pool.shutdown(); + assertTrue(pool.awaitTermination(60, TimeUnit.SECONDS)); + } finally { + graph.close(); + } + + // the lock must have kept persist() strictly single-threaded + assertEquals("commits entered persist() concurrently", 0, ConcurrencyProbeStorage.concurrentEntries.get()); + assertEquals("more than one thread was inside persist() at once", 1, ConcurrencyProbeStorage.maxObserved.get()); + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCrashConsistencyTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCrashConsistencyTest.java new file mode 100644 index 00000000000..815fb5bbb35 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCrashConsistencyTest.java @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.Graph; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.Iterator; + +import static org.junit.Assert.assertEquals; + +/** + * Crash-consistency tests for {@link GraphBinaryStorage}. Rather than kill a JVM mid-operation — which is slow and + * non-deterministic — these reconstruct the exact on-disk states a crash would leave at each step of the two durable + * sequences (the write-ahead commit and compaction) and assert that reopening recovers the correct graph. The + * invariant under test: at no step may a crash leave the store unable to recover the last committed state. + */ +public class StorageCrashConsistencyTest { + + @Rule + public TemporaryFolder tempFolder = new TemporaryFolder(); + + private String location; + private File snapshotFile; + private File logFile; + private File tmpSnapshotFile; + + @Before + public void setUp() throws Exception { + final File dir = tempFolder.newFolder("storage"); + location = dir.getAbsolutePath(); + snapshotFile = new File(dir, GraphBinaryStorage.SNAPSHOT_FILE); + logFile = new File(dir, GraphBinaryStorage.LOG_FILE); + tmpSnapshotFile = new File(dir, GraphBinaryStorage.SNAPSHOT_FILE + ".tmp"); + } + + private Configuration config() { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(Graph.GRAPH, TinkerStorageGraph.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, location); + // disable auto-compaction so tests control exactly when compaction happens + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, 0L); + return conf; + } + + private TinkerStorageGraph open() { + return TinkerStorageGraph.open(config()); + } + + /** + * Captured building blocks of valid on-disk files, used to assemble crash states: a snapshot holding {1}, a log + * holding a later commit of {2}, and a compacted snapshot holding {1,2}. + */ + private byte[] snapshotV1; + private byte[] logV2; + private byte[] snapshotV12; + + private void captureBuildingBlocks() throws IOException { + // snapshot holding vertex 1 (compaction on close folds the single commit into the snapshot) + TinkerStorageGraph g = open(); + g.addVertex(T.id, 1, "value", 1); + g.tx().commit(); + g.close(); + snapshotV1 = Files.readAllBytes(snapshotFile.toPath()); + + // a valid log holding a later commit of vertex 2, captured before the closing compaction folds it away + g = open(); // replays {1} from the snapshot + g.addVertex(T.id, 2, "value", 2); + g.tx().commit(); + g.tx().close(); + logV2 = Files.readAllBytes(logFile.toPath()); + g.close(); // compacts {1,2} into the snapshot + snapshotV12 = Files.readAllBytes(snapshotFile.toPath()); + + // reset the directory to a clean slate for the state under test + resetFiles(); + } + + private void resetFiles() throws IOException { + Files.deleteIfExists(snapshotFile.toPath()); + Files.deleteIfExists(logFile.toPath()); + Files.deleteIfExists(tmpSnapshotFile.toPath()); + } + + private void assertReopensTo(final int... expectedIds) { + final TinkerStorageGraph graph = open(); + try { + assertEquals(expectedIds.length, countOf(graph.vertices())); + for (final int id : expectedIds) + assertEquals(Integer.valueOf(id), graph.vertices(id).next().value("value")); + } finally { + graph.close(); + } + } + + @Test + public void shouldRecoverDurableCommitWithNoSnapshot() throws Exception { + // WAL guarantee: a commit durably written to the log with no compaction (no snapshot) recovers on reopen. On + // a fresh store the dictionary starts empty, so the commit frame is self-contained (it carries its own + // dictionary appends), which is exactly the real "log only, never compacted" case. + final TinkerStorageGraph g = open(); + g.addVertex(T.id, 2, "value", 2); + g.tx().commit(); + g.tx().close(); + final byte[] selfContainedLog = Files.readAllBytes(logFile.toPath()); + g.close(); // compaction on close would fold the log away; the raw log was captured above + resetFiles(); + Files.write(logFile.toPath(), selfContainedLog); + assertReopensTo(2); + } + + @Test + public void shouldRecoverFromSnapshotPlusLog() throws Exception { + // steady pre-compaction state: a snapshot holding earlier commits and a log holding later ones. Reopen must + // fold snapshot-then-log into the union. + captureBuildingBlocks(); + Files.write(snapshotFile.toPath(), snapshotV1); + Files.write(logFile.toPath(), logV2); + assertReopensTo(1, 2); + } + + @Test + public void shouldIgnoreStrayTempSnapshotFromCrashBeforeRename() throws Exception { + // crash after writing snapshot.gbin.tmp but before the atomic rename: the old snapshot + log are intact and + // the stray .tmp must be ignored, so the last committed state still recovers. + captureBuildingBlocks(); + Files.write(snapshotFile.toPath(), snapshotV1); + Files.write(logFile.toPath(), logV2); + Files.write(tmpSnapshotFile.toPath(), new byte[]{ 0x00, 0x01, 0x02, 0x03 }); // garbage half-written temp + assertReopensTo(1, 2); + } + + @Test + public void shouldRecoverFromNewSnapshotWithLogNotYetDeleted() throws Exception { + // crash after the rename installed the new snapshot but before the log was truncated: snapshot holds {1,2} + // and the stale log still holds {2}. Folding snapshot-then-log is idempotent (last write per id wins), so the + // result is exactly {1,2} — never a lost or duplicated element. + captureBuildingBlocks(); + Files.write(snapshotFile.toPath(), snapshotV12); + Files.write(logFile.toPath(), logV2); + assertReopensTo(1, 2); + } + + @Test + public void shouldRecoverCrashWindowWithADeadKeyForcingDictionaryDivergence() throws Exception { + // Guards the preserve-dictionary-numbering decision. A key ("alpha") is deleted from the live graph after it + // is in the dictionary, then a new key ("gamma") is added. Preserving numbering keeps alpha's id forever, so + // the surviving log's gamma ref still matches the new snapshot's dictionary. Renumbering on compaction would + // instead drop the now-dead alpha and shift gamma to a lower id, so the log's higher-numbered gamma ref would + // no longer resolve. A single-key (or no-delete) state cannot tell the two apart. + final TinkerStorageGraph g = open(); + g.addVertex(T.id, 1, "alpha", 1); + g.tx().commit(); + g.addVertex(T.id, 2, "beta", 2); + g.tx().commit(); + g.compact(); // snapshot holds alpha and beta in the dictionary at stable ids + + g.vertices(1).next().remove(); // alpha becomes a dead key: retained only under preserve-numbering + g.tx().commit(); + // re-write the surviving vertex: its record now carries a bare reference to the pre-existing key "beta" + // (not re-appended) plus a new key "gamma". If compaction renumbered, "beta"'s id would shift and this bare + // reference would resolve to the wrong key on replay. + g.vertices(2).next().property("gamma", "g"); + g.tx().commit(); + final byte[] logNotYetDeleted = Files.readAllBytes(logFile.toPath()); + g.compact(); // new full-dictionary snapshot (preserved numbering), then log truncated + final byte[] newSnapshot = Files.readAllBytes(snapshotFile.toPath()); + g.close(); + + // reconstruct the crash window: new snapshot in place, old log not yet deleted + Files.write(snapshotFile.toPath(), newSnapshot); + Files.write(logFile.toPath(), logNotYetDeleted); + + final TinkerStorageGraph reopened = open(); + try { + assertEquals(1, countOf(reopened.vertices())); // only the surviving vertex 2 + assertEquals(0, countOf(reopened.vertices(1))); // alpha's vertex was deleted + final Vertex v = reopened.vertices(2).next(); + assertEquals(Integer.valueOf(2), v.value("beta")); + assertEquals("g", v.value("gamma")); + } finally { + reopened.close(); + } + } + + private static long countOf(final Iterator it) { + long count = 0; + while (it.hasNext()) { + it.next(); + count++; + } + return count; + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/SyncModeTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/SyncModeTest.java new file mode 100644 index 00000000000..be1ddd2607a --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/SyncModeTest.java @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.junit.Test; + +import static org.junit.Assert.assertEquals; + +public class SyncModeTest { + + @Test + public void shouldDefaultToCommitWhenUnset() { + assertEquals(SyncMode.COMMIT, SyncMode.fromConfigValue(null)); + } + + @Test + public void shouldResolveCaseInsensitivelyAndTrim() { + assertEquals(SyncMode.COMMIT, SyncMode.fromConfigValue("commit")); + assertEquals(SyncMode.COMMIT, SyncMode.fromConfigValue("COMMIT")); + assertEquals(SyncMode.OS, SyncMode.fromConfigValue("os")); + assertEquals(SyncMode.OS, SyncMode.fromConfigValue(" Os ")); + } + + @Test(expected = IllegalArgumentException.class) + public void shouldRejectUnknownValue() { + SyncMode.fromConfigValue("interval:1000"); + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerByteBufferTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerByteBufferTest.java new file mode 100644 index 00000000000..006adba8c8d --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerByteBufferTest.java @@ -0,0 +1,243 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.junit.Test; + +import java.io.ByteArrayOutputStream; +import java.nio.ByteBuffer; +import java.util.Arrays; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * Unit tests for {@link TinkerByteBuffer}, the heap {@code byte[]} implementation of the gremlin-core + * {@code Buffer} abstraction that backs the GraphBinary storage codec. The byte order and bounds behaviour asserted + * here are relied on by the on-disk format, so a change that breaks them changes what a store means. + */ +public class TinkerByteBufferTest { + + @Test + public void shouldRoundTripEveryPrimitive() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeBoolean(true).writeBoolean(false) + .writeByte(0x7F).writeShort(-2) + .writeInt(Integer.MIN_VALUE).writeLong(Long.MAX_VALUE) + .writeFloat(0.5f).writeDouble(-1.25d); + + assertTrue(buf.readBoolean()); + assertFalse(buf.readBoolean()); + assertEquals(0x7F, buf.readByte()); + assertEquals((short) -2, buf.readShort()); + assertEquals(Integer.MIN_VALUE, buf.readInt()); + assertEquals(Long.MAX_VALUE, buf.readLong()); + assertEquals(0.5f, buf.readFloat(), 0.0f); + assertEquals(-1.25d, buf.readDouble(), 0.0d); + assertEquals(0, buf.readableBytes()); + } + + @Test + public void shouldWriteMultiByteValuesBigEndian() { + // the on-disk format is big-endian, matching the Netty-backed buffer this replaces, so the byte order is + // part of the storage contract rather than an implementation detail + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeShort(0x0102).writeInt(0x03040506).writeLong(0x0708090A0B0C0D0EL); + + assertArrayEquals(new byte[] { + 0x01, 0x02, + 0x03, 0x04, 0x05, 0x06, + 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E + }, buf.toWrittenArray()); + } + + @Test + public void shouldGrowBeyondInitialCapacity() { + final TinkerByteBuffer buf = new TinkerByteBuffer(1); + final byte[] payload = new byte[1000]; + Arrays.fill(payload, (byte) 7); + buf.writeBytes(payload); + + assertTrue("capacity should have grown to hold the payload", buf.capacity() >= 1000); + assertEquals(1000, buf.readableBytes()); + assertArrayEquals(payload, buf.toWrittenArray()); + } + + @Test + public void shouldWrapAnExistingArrayReadyForReading() { + final byte[] data = { 0x00, 0x00, 0x00, 0x2A }; + final TinkerByteBuffer buf = new TinkerByteBuffer(data); + + assertEquals(4, buf.writerIndex()); + assertEquals(0, buf.readerIndex()); + assertEquals(4, buf.readableBytes()); + assertEquals(42, buf.readInt()); + } + + @Test + public void shouldTrackReaderAndWriterIndexes() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeInt(1).writeInt(2); + assertEquals(8, buf.writerIndex()); + assertEquals(8, buf.readableBytes()); + + buf.readInt(); + assertEquals(4, buf.readerIndex()); + assertEquals(4, buf.readableBytes()); + + buf.readerIndex(0); + assertEquals(1, buf.readInt()); + } + + @Test + public void shouldDistinguishReadableFromWrittenBytes() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeInt(1).writeInt(2); + buf.readInt(); + + // written covers everything from index 0; readable covers only what is left ahead of the reader + assertArrayEquals(new byte[] { 0, 0, 0, 1, 0, 0, 0, 2 }, buf.toWrittenArray()); + assertArrayEquals(new byte[] { 0, 0, 0, 2 }, buf.toReadableArray()); + assertEquals("neither view may move the indexes", 4, buf.readerIndex()); + } + + @Test + public void shouldResetWriterIndexToTheMark() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeInt(1); + buf.markWriterIndex(); + buf.writeInt(2); + assertEquals(8, buf.writerIndex()); + + buf.resetWriterIndex(); + assertEquals(4, buf.writerIndex()); + assertArrayEquals(new byte[] { 0, 0, 0, 1 }, buf.toWrittenArray()); + } + + @Test + public void shouldRejectReadingPastTheWriterIndex() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeByte(1); + buf.readByte(); + + try { + buf.readByte(); + fail("expected a read past the writer index to be rejected"); + } catch (IndexOutOfBoundsException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("Not enough readable bytes")); + } + } + + @Test + public void shouldRejectReadingMoreBytesThanRemain() { + final TinkerByteBuffer buf = new TinkerByteBuffer(new byte[] { 1, 2, 3 }); + try { + buf.readBytes(new byte[4]); + fail("expected a bulk read longer than the readable region to be rejected"); + } catch (IndexOutOfBoundsException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("Not enough readable bytes")); + } + } + + @Test + public void shouldRejectAnOutOfRangeReaderIndex() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeInt(1); + try { + buf.readerIndex(5); + fail("expected a reader index beyond the writer index to be rejected"); + } catch (IndexOutOfBoundsException expected) { + // expected: the readable region may never extend past what has been written + } + } + + @Test + public void shouldReadBytesIntoAnArraySlice() { + final TinkerByteBuffer buf = new TinkerByteBuffer(new byte[] { 1, 2, 3, 4 }); + final byte[] dst = new byte[6]; + buf.readBytes(dst, 1, 4); + + assertArrayEquals(new byte[] { 0, 1, 2, 3, 4, 0 }, dst); + assertEquals(0, buf.readableBytes()); + } + + @Test + public void shouldReadAndWriteThroughNioBuffers() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeBytes(ByteBuffer.wrap(new byte[] { 9, 8, 7 })); + assertEquals(3, buf.readableBytes()); + + final ByteBuffer dst = ByteBuffer.allocate(3); + buf.readBytes(dst); + assertArrayEquals(new byte[] { 9, 8, 7 }, dst.array()); + } + + @Test + public void shouldReadBytesIntoAnOutputStream() throws Exception { + final TinkerByteBuffer buf = new TinkerByteBuffer(new byte[] { 4, 5, 6, 7 }); + final ByteArrayOutputStream out = new ByteArrayOutputStream(); + buf.readBytes(out, 3); + + assertArrayEquals(new byte[] { 4, 5, 6 }, out.toByteArray()); + assertEquals(1, buf.readableBytes()); + } + + @Test + public void shouldCopyAbsoluteBytesWithoutMovingIndexes() { + final TinkerByteBuffer buf = new TinkerByteBuffer(new byte[] { 1, 2, 3, 4 }); + final byte[] dst = new byte[2]; + buf.getBytes(2, dst); + + assertArrayEquals(new byte[] { 3, 4 }, dst); + assertEquals("an absolute read is positional and must not consume", 0, buf.readerIndex()); + } + + @Test + public void shouldExposeAnNioViewOfTheReadableRegion() { + final TinkerByteBuffer buf = new TinkerByteBuffer(new byte[] { 1, 2, 3, 4 }); + buf.readByte(); + + assertEquals(1, buf.nioBufferCount()); + final ByteBuffer view = buf.nioBuffer(); + assertArrayEquals(new byte[] { 2, 3, 4 }, view.array()); + assertEquals("the view is a copy, so consuming it must not move the buffer", 3, buf.readableBytes()); + assertArrayEquals(new byte[] { 2, 3 }, buf.nioBuffer(1, 2).array()); + assertEquals(1, buf.nioBuffers().length); + } + + @Test + public void shouldCountReferences() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + assertEquals(1, buf.referenceCount()); + + buf.retain(); + assertEquals(2, buf.referenceCount()); + assertFalse("still referenced, so release does not report the last one", buf.release()); + assertEquals(1, buf.referenceCount()); + assertTrue("the final release reports that the buffer is done", buf.release()); + } + + @Test + public void shouldReportAsHeapBacked() { + assertFalse(new TinkerByteBuffer().isDirect()); + } +}