nbdkit-multi-conn-filter - enable, emulate or disable multi-conn
nbdkit --filter=multi-conn plugin
[multi-conn-mode=MODE] [multi-conn-track-dirty=LEVEL]
[multi-conn-exportname=BOOL]
[plugin-args...]NBD_FLAG_CAN_MULTI_CONN ("multi-conn") is an NBD protocol feature that permits multiple clients to connect to the same export simultaneously, guaranteeing that flush operations are consistent across connections. Specifically a sequence of getting a write response, sending and waiting for a flush response, then sending a read request will behave the same whether all three commands shared a single connection or were split among three connections. When an NBD client and server are able to negotiate this feature it can provide significant performance benefits. Conversely if the feature is not advertised, clients must presume that separate connections can cache writes independently (so even after waiting for a flush on one connection, a read on another connection may see stale data from a cache). The NBD standard advises clients not to multiplex commands across connections if the server does not support multi-conn.
nbdkit(1) plugins must normally opt in to multi-conn, after carefully ensuring the implementation meets the consistency requirements. This filter can emulate flush-consistent semantics across multiple connections for plugins that do not advertise this feature.
This filter also has additional modes useful for evaluating performance and correctness of client and plugin multi-conn behaviors.
This filter assumes that multiple connections to a plugin will eventually share data, other than any caching effects. It is not suitable for use with a plugin that produces completely independent data per connection from the same export name. An example of a plugin that must not be used with this filter is nbdkit-tmpdisk-plugin(1).
Additional control over the behavior of client flush commands is possible by combining this filter with nbdkit-fua-filter(1). Note that nbdkit-cache-filter(1) is also able to provide multi-connection flush consistency, but at the expense of an extra layer of caching not needed with this filter.
This is the default mode. The behaviour of auto is as follows:
If the selected thread model is SERIALIZE_CONNECTIONS, then this filter behaves the same as disable mode.
If the plugin advertises multi-conn, then this filter behaves the same as plugin mode.
Otherwise, this filter behaves the same as emulate mode.
In other words, this mode advertises multi-conn to the client exactly when the plugin supports or can be made to support multiple simultaneous connections.
When emulate mode is chosen, then this filter tracks all parallel connections. When a client issues a flush command over any one connection (including an implied flush by a write command with the FUA (force unit access) flag set), the filter then replicates that flush across each connection to the plugin. The number of plugin calls made by the filter can be tuned by adjusting multi-conn-track-dirty.
This mode assumes that flushing each connection is enough to clear any per-connection cached data, in order to give each connection a consistent view of the image; therefore, this mode advertises multi-conn to the client.
Note that in this mode, a client will be unable to connect if the plugin lacks support for flush, as there would be no way to emulate cross-connection flush consistency.
When disable mode is chosen, this filter disables advertisement of multi-conn to the client, even if the plugin supports it, and does not replicate flush commands across connections. This is useful for testing whether a client with multiple connections properly sends multiple flushes in order to overcome per-connection caching.
When plugin mode is chosen, the filter does not change whether multi-conn is advertised by the plugin, and does not replicate flush commands across connections; but still honors multi-conn-track-dirty for minimizing the number of flush commands passed on to the plugin.
When unsafe mode is chosen, this filter blindly advertises multi-conn to the client even if the plugin lacks support. This is dangerous, and risks data corruption if the client makes assumptions about flush consistency that the plugin does not actually provide. However, for a plugin that does not yet advertise multi-conn, but where it is suspected that the plugin behaves consistently, this is a great way to run timing and accuracy tests to see whether enabling multi-conn in the plugin will make a difference.
When dirty tracking is set to fast, the filter tracks whether any connection has caused the image to be dirty (any write, zero, or trim commands since the last flush, regardless of connection); if all connections are clean, a client flush command is ignored rather than sent on to the plugin. In this mode, a flush action on one connection marks all other connections as clean, regardless of whether the filter actually advertised multi-conn, which can result in less activity when a client sends multiple flushes rather than taking advantage of multi-conn semantics. This is safe with multi-conn-mode=emulate, but potentially unsafe with multi-conn-mode=plugin when the plugin did not advertise multi-conn, as it does not track whether a read may have cached stale data prior to a flush.
This is the default setting for multi-conn-track-dirty.
The filter tracks whether a given connection is dirty (any write, zero, or trim commands since the last flush on the given connection, and any read since the last flush on any other connection); if the connection is clean, a flush command to that connection (whether directly from the client, or replicated by multi-conn-mode=emulate is ignored rather than sent on to the plugin. This mode may result in more flush calls than multi-conn-track-dirty=fast, but in turn is safe to use with multi-conn-mode=plugin.
When dirty tracking is set to off, all flush commands from the client are passed on to the plugin, regardless of whether the flush would be needed for cross-connection consistency. Note that when combined with multi-conn-mode=emulate, a client which disregards multi-conn by flushing on each connection itself results in a quadratic number of flush operations on the plugin.
The exportname switch defaults to false for safety, and causes the filter to flush across all active connections regardless of the export name in use by that connection when doing emulation. However, when a plugin supports distinct data according to export name, this behavior will penalize the performance of clients visiting an unrelated export by spending time on replicated flush operations not actually relevant to that export.
Setting the exportname switch to true causes the filter to only synchronize flushes to connections visiting the same export name. This avoids penalizing clients visiting an unrelated export name (such as nbdkit-file-plugin(1) in dir= mode), but is unsafe when used with a plugin that serves shared content across all connections regardless of the export name requested by the client, if that plugin is not already multi-conn consistent (such as nbdkit-vddk-plugin(1)).
Provide consistent cross-connection flush semantics on top of a plugin that lacks it natively:
nbdkit --filter=multi-conn vddk /absolute/path/to/file.vmdkMinimize the number of expensive flush operations performed when utilizing a plugin that has multi-conn consistency from a client that blindly flushes across every connection:
nbdkit --filter=multi-conn file multi-conn-mode=plugin \
multi-conn-track-dirty=fast disk.imgThe filter.
Use nbdkit --dump-config to find the location of $filterdir.
nbdkit-multi-conn-filter first appeared in nbdkit 1.26.
nbdkit(1), nbdkit-file-plugin(1), nbdkit-vddk-plugin(1), nbdkit-filter(3), nbdkit-cache-filter(1), nbdkit-fua-filter(1), nbdkit-nocache-filter(1), nbdkit-noextents-filter(1), nbdkit-noparallel-filter(1), nbdkit-nozero-filter(1), https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md
Eric Blake
Copyright Red Hat
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of Red Hat nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.