Initial work on ReindexDatastreamIndexAction #116996
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add an action which reindexes a single data stream index. Most of the work involves settings the correct settings before and after the reindexing. Eventually, this action will handle reindexing searchable snapshot indices, but this is left out for now.
masseyke
reviewed
Nov 20, 2024
...rc/main/java/org/elasticsearch/datastreams/action/TransportReindexDataStreamIndexAction.java
Outdated
Show resolved
Hide resolved
masseyke
reviewed
Nov 20, 2024
...rc/main/java/org/elasticsearch/datastreams/action/TransportReindexDataStreamIndexAction.java
Outdated
Show resolved
Hide resolved
elasticsearchmachine
added
the
needs:triage
parkertimmins
added
auto-backport
|
Pinging @elastic/es-data-management (Team:Data Management) |
elasticsearchmachine
added
the
Team:Data Management
|
Hi @parkertimmins, I've created a changelog YAML for you. |
|
Seems fine for an initial pass, once you put the action in OperatorPrivilegesIT. Nothing will actually use this yet, so it's just a placeholder. |
masseyke
approved these changes
Nov 21, 2024
masseyke
reviewed
Nov 21, 2024
server/src/main/java/org/elasticsearch/action/datastreams/ReindexDataStreamIndexAction.java
Outdated
Show resolved
Hide resolved
parkertimmins
commented
Dec 11, 2024
| .addListener(listener); | ||
| } | ||
|
|
||
| private void setBlockWrites(String sourceIndexName, ActionListener<AcknowledgedResponse> listener) { |
There was a problem hiding this comment.
Should we remove the write block on the source index after reindexing? I'm inclined to say it's fine to leave it, but if we failed/cancelled an instance of this call then reran it, the block-writes setting would then be found on the source index and copied to the new dest index.
There was a problem hiding this comment.
We discussed adding a flag to the cluster state which says whether an index had block-write before the call to this action. This would then be passed as a param into the action.
masseyke
reviewed
Dec 11, 2024
.../main/java/org/elasticsearch/xpack/migrate/action/ReindexDataStreamIndexTransportAction.java
Outdated
Show resolved
Hide resolved
masseyke
reviewed
Dec 11, 2024
.../main/java/org/elasticsearch/xpack/migrate/action/ReindexDataStreamIndexTransportAction.java
Outdated
Show resolved
Hide resolved
masseyke
reviewed
Dec 11, 2024
.../main/java/org/elasticsearch/xpack/migrate/action/ReindexDataStreamIndexTransportAction.java
Outdated
Show resolved
Hide resolved
masseyke
reviewed
Dec 11, 2024
...grate/src/main/java/org/elasticsearch/xpack/migrate/action/ReindexDataStreamIndexAction.java
Outdated
Show resolved
Hide resolved
masseyke
reviewed
Dec 13, 2024
| assertEquals(endTime, destEnd); | ||
| } | ||
|
|
||
| // TODO more logsdb/tsdb specific tests |
There was a problem hiding this comment.
There was no logsdb/tsdb in 7.x was there? So nothing to upgrade?
There was a problem hiding this comment.
Oh sweet. Well, then I can remove some tests
💔 Backport failed
You can use sqren/backport to manually backport by running |
💚 All backports created successfully
Questions ?Please refer to the Backport tool documentation |
parkertimmins
added a commit
to parkertimmins/elasticsearch
that referenced
this pull request
Dec 13, 2024
Add an action to reindex a single index from a source index to a destination index. Unlike the reindex action, this action copies settings and mappings from the source index to the dest index before performing the reindex. This action is part of work to reindex data streams and will be called on each of the backing indices within a data stream. (cherry picked from commit 0a6ce27)
parkertimmins
added a commit
that referenced
this pull request
Dec 13, 2024
Add an action to reindex a single index from a source index to a destination index. Unlike the reindex action, this action copies settings and mappings from the source index to the dest index before performing the reindex. This action is part of work to reindex data streams and will be called on each of the backing indices within a data stream. (cherry picked from commit 0a6ce27)
maxhniebergall
pushed a commit
to maxhniebergall/elasticsearch
that referenced
this pull request
Dec 16, 2024
Add an action to reindex a single index from a source index to a destination index. Unlike the reindex action, this action copies settings and mappings from the source index to the dest index before performing the reindex. This action is part of work to reindex data streams and will be called on each of the backing indices within a data stream. (cherry picked from commit 0a6ce27)
This was referenced Dec 17, 2024
parkertimmins
added a commit
that referenced
this pull request
Jan 16, 2025
When creating a destination index, filter any deprecated settings before copying the source settings to the dest index. Specifically, filter out settings with the property IndexSettingDeprecatedInV7AndRemovedInV8 or settings from a specific list which do not have this property but have no effect in v8. This also includes a change which reverts the changes that #116996 made to MetadataCreateIndexService. That change factored the method copySettingsFromSource out of prepareResizeIndexSettings so it could be used to filter settings in CreateIndexFromSourceAction. The additional filtering of deprecated property caused the requirements in CreateIndexFromSourceAction and MetadataCreateIndexService to diverge. So this change reverts the original refactor to MetadataCreateIndexService and adds a method similar to copySettingsFromSource to CreateIndexFromSourceAction.
parkertimmins
added a commit
to parkertimmins/elasticsearch
that referenced
this pull request
Jan 16, 2025
…0163) When creating a destination index, filter any deprecated settings before copying the source settings to the dest index. Specifically, filter out settings with the property IndexSettingDeprecatedInV7AndRemovedInV8 or settings from a specific list which do not have this property but have no effect in v8. This also includes a change which reverts the changes that elastic#116996 made to MetadataCreateIndexService. That change factored the method copySettingsFromSource out of prepareResizeIndexSettings so it could be used to filter settings in CreateIndexFromSourceAction. The additional filtering of deprecated property caused the requirements in CreateIndexFromSourceAction and MetadataCreateIndexService to diverge. So this change reverts the original refactor to MetadataCreateIndexService and adds a method similar to copySettingsFromSource to CreateIndexFromSourceAction.
elasticsearchmachine
pushed a commit
that referenced
this pull request
Jan 16, 2025
…120319) When creating a destination index, filter any deprecated settings before copying the source settings to the dest index. Specifically, filter out settings with the property IndexSettingDeprecatedInV7AndRemovedInV8 or settings from a specific list which do not have this property but have no effect in v8. This also includes a change which reverts the changes that #116996 made to MetadataCreateIndexService. That change factored the method copySettingsFromSource out of prepareResizeIndexSettings so it could be used to filter settings in CreateIndexFromSourceAction. The additional filtering of deprecated property caused the requirements in CreateIndexFromSourceAction and MetadataCreateIndexService to diverge. So this change reverts the original refactor to MetadataCreateIndexService and adds a method similar to copySettingsFromSource to CreateIndexFromSourceAction.
rjernst
pushed a commit
to rjernst/elasticsearch
that referenced
this pull request
Jan 16, 2025
…0163) When creating a destination index, filter any deprecated settings before copying the source settings to the dest index. Specifically, filter out settings with the property IndexSettingDeprecatedInV7AndRemovedInV8 or settings from a specific list which do not have this property but have no effect in v8. This also includes a change which reverts the changes that elastic#116996 made to MetadataCreateIndexService. That change factored the method copySettingsFromSource out of prepareResizeIndexSettings so it could be used to filter settings in CreateIndexFromSourceAction. The additional filtering of deprecated property caused the requirements in CreateIndexFromSourceAction and MetadataCreateIndexService to diverge. So this change reverts the original refactor to MetadataCreateIndexService and adds a method similar to copySettingsFromSource to CreateIndexFromSourceAction.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add an action which reindexes a single data stream index. The contents of this action are not yet complete, but provide a way to wire into the reindex datastream persistent task. Eventually this action will do the following: