Repository navigation
Conversation
Introduce the WalProvider SPI and properties to configure the available set of WAL providers and the current provider. This commit converts the current DFS implementation to a DfsWalProvider and the upgrade code converts all existing wal entries in ZooKeeper and the root and metadata tables to use the 'dfs' provider id. Post-upgrade and on instance restart the `instance.wal.provider' property can be set to the class name of a WALProvider SPI implementation. The 'instance.wal.providers' property must contain the class name and configuration properties for the current and all in-use providers. An InMemoryWalProvider class in the test module provides an example implementation and is used in various ITs. Assisted-By: OpenAI GPT-6 Luna
There was a problem hiding this comment.
This file will be removed before merge and moved over to the website repo
keith-turner
left a comment
keith-turner
left a comment
There was a problem hiding this comment.
Took a quick look at the SPI, have not looked at the impl changes yet. Will do that soon.
| * Returns the current replication level when the provider can report one, or {@code -1} when | ||
| * that information is unavailable. | ||
| */ | ||
| default int getCurrentReplication() throws IOException { |
There was a problem hiding this comment.
The way this is used is to determine if a wal file is healthy and if its not it abandons it and starts a new one. Could make this method more generic.
| default int getCurrentReplication() throws IOException { | |
| default boolean isHealthy() throws IOException { |
| * Returns a diagnostic description of the current storage pipeline, or an empty array when the | ||
| * provider has no pipeline information. | ||
| */ | ||
| default String[] getPipeline() { |
There was a problem hiding this comment.
Could make this more generic and avoid the specifics of HDFS impl.
| default String[] getPipeline() { | |
| default String getDiagnostics() throws IOException { |
| * {@link #sync(Durability)} while other threads append through the output stream; implementations | ||
| * must ensure a sync covers all bytes written before it is requested. | ||
| */ | ||
| interface Writer extends AutoCloseable { |
There was a problem hiding this comment.
This may not work well at all, so not suggesting we following this path was just curious about this approach. Was wondering if instead of having a getOutputStream() method we could extend OutputStream instead
| interface Writer extends AutoCloseable { | |
| interface Writer extends OutputStream { |
Again not recommending we do this, not sure how it would impact the code if it would be an improvement or not. Was wondering if you had a sense of how it would play out in these changes and if it would be beneficial or not.
|
@keith-turner - before looking at this further, take a look at #6601. That should probably be merged first and will help isolate some of the changes in this PR a little better. |
This branch has not been deployed
2 participants
Introduce the WalProvider SPI and properties to configure the available set of WAL providers and the current provider.
This commit converts the current DFS implementation to a DfsWalProvider and the upgrade code converts all existing wal entries in ZooKeeper and the root and metadata tables to use the 'dfs' provider id.
Post-upgrade and on instance restart the
`instance.wal.provider' property can be set to the class name of a WALProvider SPI implementation. The
'instance.wal.providers' property must contain the class name and configuration properties for the current and all in-use providers.
An InMemoryWalProvider class in the test module provides an example implementation and is used in various ITs.
Assisted-By: OpenAI GPT-6 Luna