Upgrading Elastic from 7.14.0 to 8.0.0

Earlier versions of the starter dashboards were developed in Elastic 7.14.0. The current dashboards were developed in Elastic 8.0.0.

8.0.0 is the minimum Elastic version required for the current dashboards, and also the recommended version, to avoid possible compatibility issues.

To upgrade pre-8.0.0 Elastic environments that you installed and configured yourself, see the Elastic Stack documentation.

To upgrade Elastic 7.14.0 environments that you deployed using the Docker Compose file supplied in previous versions of this repository, continue reading.

Upgrading your Docker Compose environment

The Docker Compose environment uses a named volume to store indexed Elasticsearch data, including Kibana spaces, dashboards, and related saved objects.

The current Docker Compose file refers to the same named volume, but starts an Elastic 8.0.0 environment. The version difference between the 7.14.0 data artifacts and the 8.0.0 application stack introduces compatibility issues.

The old dashboards aren't compatible with Elastic 8.0.0. You need to delete them.

Important

If you have customized or created your own dashboards in the Docker Compose environment, and you want to keep those dashboards, then you are responsible for doing that, and the remaining instructions presented here do not apply. It's your choice how and whether to migrate those dashboards to a more recent Elastic version.

It's your choice whether to delete or keep old data. Keeping old data involves more steps than deleting it.

Get the current repository

Copy the current GitHub repository z-open-data/odp-elastic-samples to your Docker host.

Delete or keep old data

Decide which option is right for you, and then perform one of the following procedures.

Relative paths in the following procedures are relative to the repository root directory.

Delete old data

Tear down the old Elastic 7.14.0-based Docker Compose environment.
Change to the samples/docker/odp-elastic directory. Enter the following command:
```
docker compose down
```
Delete the named volume.
Warning
The following command deletes all indexed Elasticsearch data, including Kibana spaces, dashboards, and related saved objects.
Enter the following command:
```
docker volume rm odp-elastic_elastic-data
```
The next time that you deploy the Docker Compose environment, the environment creates an empty named volume.
Deploy the Elastic 8.0.0-based Docker Compose environment.
See Deploying Elastic using the provided Docker Compose file.

Keep old data

If you want to keep old data indexed by a pre-8.0.0 Elastic environment, then you need to upgrade that pre-8.0.0 Elastic environment.

The upgrade involves multiple steps, because Elasticsearch does not allow you to upgrade directly from 7.14.0 to 8.0.0. Instead, you must upgrade to 7.17.0 (the last 7.x.x), and then to 8.0.0.

Stop incoming data.
For example, stop instances of OMEGAMON Data Connect that send data to Logstash in the Docker Compose environment.
Stop the old (Elastic 7.14.0-based) Docker Compose environment.
Change to the samples/docker/odp-elastic directory.
Enter the following command:
```
docker compose stop
```
Optional (but recommended): Back up the old data in a new Docker volume; as a precaution, in case anything goes wrong during the upgrade.
Create an empty backup volume:
```
docker volume create odp-elastic_elastic-data-backup
```
where odp-elastic_elastic-data-backup is your choice of name for the backup volume.
Copy data to the backup volume. For example:
```
docker run --rm -v odp-elastic_elastic-data:/from -v odp-elastic_elastic-data-backup:/to alpine sh -c "cp -av /from/. /to/"
```
Start the old Docker Compose environment:
```
docker compose start
```
Delete the old dashboards and their related saved objects.
For example, use Kibana to perform one of the following actions (your choice):
- Delete all of the saved objects in the Kibana spaces that contain the dashboards.
- Delete the Kibana spaces. This is quicker, but means that you need to re-create spaces later, before installing the current dashboards.
Tear down the old Docker Compose environment:
```
docker compose down
```
Set the Elastic version to 7.17.0.
Edit the samples/docker/odp-elastic/.env file and change the Elastic Stack version from 8.0.0 to 7.17.0:
```
STACK_VERSION=7.17.0
```
Deploy the Elastic 7.17.0-based Docker Compose environment:
```
docker compose up -d
```
Before proceeding to the next step, inspect the Elastic Stack logs and verify that all services are up and running.
Set the number of replicas for deprecation log indices to 0.
From version 7.16, Elasticsearch enables deprecation logs by default, creating indices with number_of_replicas: 1. However, the Docker Compose file starts a single-node Elasticsearch cluster. There are no nodes available for replicas, so the cluster health status is yellow.
Check the Elasticsearch cluster health status by using the Kibana Dev Tools console to send the following request:
```
GET _cat/health?v=true
```
To fix this issue, use the Kibana Dev Tools console to set all existing deprecation log indices to number_of_replicas: 0:
```
PUT /.ds-.logs-deprecation.elasticsearch-default-*/_settings
{
  "index": {
    "number_of_replicas": 0
  }
}
```
Edit the related component template to set number_of_replicas: 0 on future deprecation log indices.
In Kibana, select Home > Management: Stack Management > Data: Index Management > Component Templates. Select the .deprecation-indexing-settings template, and then click Edit. On the Index settings step, insert "number_of_replicas": 0 under "index":
```
{
  "index": {
    "number_of_replicas": 0,
     ...
  }
}
```
Go to the Review step and save the updated template.
Before proceeding to the next step, check that the cluster health status is now green.
Tear down the Elastic 7.17.0-based Docker Compose environment:
```
docker compose down
```
Set the Elastic version to 8.0.0.
Edit the samples/docker/odp-elastic/.env file and change the Elastic Stack version to 8.0.0:
```
STACK_VERSION=8.0.0
```
Restart incoming data.
For example, restart instances of OMEGAMON Data Connect that send data to Logstash in the Docker Compose environment.
Deploy the Elastic 8.0.0 Docker Compose environment.
See Deploying Elastic using the provided Docker Compose file.