# Migration from MongoDB to Elasticsearch

*Created by Seokje Park at 2020.02.21*

keyword

* `mongodb to elasticsearch`
* `migratino`

We will sync mongoDB with Elasticsearch

## MongoDB upgrade 3.6 to 4.0

I followed this [instructions](https://docs.mongodb.com/v4.0/tutorial/install-mongodb-on-amazon/).

If you come out as below, you have succeeded.

```bash
  $ mongo
    rs0:PRIMARY> db.version()
    4.0.16
```

## Setup Replica set on a single machine

Referenced [here](https://www.tutorialkart.com/mongodb/setup-mongodb-replica-set/).

```bash
  $ 
  $ sudo mongod --fork --logpath /var/log/mongod0.log --port 27017 --dbpath /var/lib/mongo --replSet rs0 # Start a mongod instance
  $ sudo mkdir -p /var/lib/mongodb1/ # Start another mongod instance
  $ sudo mongod --fork --logpath /var/log/mongod1.log --port 27018 --dbpath /var/lib/mongo1 --replSet rs0
  $ mongo
  $  > rs.initiate()
  $  > rs.add('localhost:27018') # { "ok"" 1 }
  $  > rs.status()
```

* `sudo killall -15 mongod`: Kill all mongod process. (Use it if you want to upgrade or restart mongod)
* `--fork --logpath /var/log/mongod0.log`: Commands to Run mongod as Daemons.
* `ps -edaf | grep mongo | grep -v grep`: Command to check the running mongod process.

## Install Golang-go

```bash
  $ sudo yum install golang
  $ go env # test
```

## Download Monstache & upzip

```bash
  $ wget -O /home/ec2-user/monstache.zip "https://github.com/rwynn/monstache/releases/download/v6.5.1/monstache-8e69a61.zip"
  $ unzip monstache.zip -d ./monstache
```

## Export your path to the .profile

```bash
  $ sudo vi ~/.basrhc
```

Add the following lines

```bash
  PATH="/home/ec2-user/monstache/build/linux-amd64:$PATH"
```

Test

```bash
  $ monstache -v
  6.5.1
```

## Setup monstache config

More options can be found [here](https://rwynn.github.io/monstache-site/start/).

`{Any path}/config.toml`

```
  mongo-url = "mongodb://localhost:27018"
  elasticsearch-urls = ["http://{elasticsearch url}:9200"]
  elasticsearch-max-conns = 4
  elasticsearch-max-seconds = 5
  elasticsearch-max-bytes = 8000000

  namespace-regex = '^{dbname}\.{colname}$'
  change-stream-namespaces  = ["dbname}.{colname}"]

  gzip = true
  stats = false
  index-stats = true

  dropped-collections = false
  dropped-databases = false

  replay = false
  resume = true
  resume-write-unsafe = false
  resume-name = "default"
  verbose = true
  exit-after-direct-reads = false
```

The above config needs futher investigation.

## Start Synchronize MonogoDB to Elasticsearch using monstache

```bash
  $ monstache -f config.toml # watch mode
  $ nohup monstache -f config.toml & # background process
```

## Security

When you apply Elasticsearch basic authentication later, you must make the following changes.

```bash
  $ sudo vi config.toml
    elasticsearch-user = "elastic"
    elasticsearch-password = "{elasticsearch-setup-passwords generated elastic password}"
```

See more about elasticsearch security [Setting Elaticsearch and Kibana](https://molla4455.gitbook.io/dev-log/elastic/elasticsearch#secure-elasticsearch-and-kibana).

## Reference site

* [Tutorial: Sync mongoDB with Elasticsearch](https://medium.com/@eldishnawy/tutorial-sync-mongodb-with-elasticsearch-fb43e9bc13ce)
* [5 Different ways to synchronize data from MongoDB to ElasticSearch](https://code.likeagirl.io/5-different-ways-to-synchronize-data-from-mongodb-to-elasticsearch-d8456b83d44f)
* [MongoDB to ElasticSearch Sync using Monstache](https://medium.com/@nehajirafe/mongodb-to-elasticsearch-sync-using-monstache-cfe1177594b6)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://molla4455.gitbook.io/dev-log/elastic/migration.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.