{"id":2602,"date":"2018-09-04T13:43:52","date_gmt":"2018-09-04T13:43:52","guid":{"rendered":"http:\/\/dbtut.com\/?p=2602"},"modified":"2018-11-24T16:47:25","modified_gmt":"2018-11-24T16:47:25","slug":"mongodb-resync-stale-replica-member","status":"publish","type":"post","link":"https:\/\/dbtut.com\/index.php\/2018\/09\/04\/mongodb-resync-stale-replica-member\/","title":{"rendered":"MongoDB Resync stale replica member"},"content":{"rendered":"
<\/p>\n
In a replica set if one of the members is in\u00a0RECOVERING\u00a0<\/span>state and in the log file of that member if you see:<\/p>\n
2018-02-14T16:02:50.410+0300 E REPL \u00a0\u00a0\u00a0\u00a0[rsBackgroundSync] too stale to catch up -- entering maintenance mode\r\n\r\n2018-02-14T16:02:50.410+0300 I REPL \u00a0\u00a0\u00a0\u00a0[rsBackgroundSync] our last optime : (term: 32, timestamp: Jan 12 22:42:48:1c51)\r\n\r\n2018-02-14T16:02:50.410+0300 I REPL \u00a0\u00a0\u00a0\u00a0[rsBackgroundSync] oldest available is (term: 38, timestamp: Jan 29 13:53:34:6f)<\/pre>\n
<\/p>\n
This means that the oplog size wasn’t enough to handle the time that the replica server was unreachable. A network problem, lack of disk space or a closed host may cause these type of problems .<\/p>\n
Oplog size must be enough during problem solving process. If the problem isn’t solved during this time, the replica will come into RECOVERING state.<\/p>\n
The only thing you can do here is resyncing the replica member from primary.<\/p>\n
<\/p>\n
There are two ways:<\/strong><\/p>\n
1- Initial sync (with no initial data)<\/p>\n
2- sync by copying data files from another member<\/p>\n
<\/p>\n
In the first method steps are:<\/strong><\/p>\n
1- shutdown the server.<\/p>\n
use admin\r\n\r\ndb.shutdownServer()<\/pre>\n
<\/p>\n
2- remove the old data file directory<\/p>\n
3- create an empty data file directory with the rights.<\/p>\n
4- start the mongod process<\/p>\n
Status of the replica will become startup2 from recovering.<\/p>\n
<\/p>\n
Second method:<\/strong><\/p>\n
During the syncronization process, primary may not answer the stale secondary and this causes restarting the syncronization process.<\/p>\n
If the primary is answering other requests at the same time and syncing process is long,\u00a0these interruptions may occur repeatedly. The syncronization never ends.<\/p>\n
In secondary log:<\/strong><\/p>\n
2018-08-28T05:17:05.817+0300 I INDEX \u00a0\u00a0\u00a0[rsSync] build index done. \u00a0scanned 138338099 total records. 5629 secs\r\n\r\n2018-08-28T05:17:05.827+0300 I STORAGE \u00a0[rsSync] copying indexes for: { name: \"Announcement\", options: {} }\r\n\r\n2018-08-28T05:17:05.827+0300 I NETWORK \u00a0[rsSync] Socket say send() errno:32 Broken pipe xxxxxxxxxxxxxxxxxxxx\r\n\r\n2018-08-28T05:17:05.889+0300 E REPL \u00a0\u00a0\u00a0\u00a0[rsSync] 9001 socket exception [SEND_ERROR] server xxxxxxxxxxxxxxxxxxxxxxx\r\n\r\n2018-08-28T05:17:05.889+0300 E REPL \u00a0\u00a0\u00a0\u00a0[rsSync] initial sync attempt failed, 9 attempts remaining\r\n\r\n2018-08-28T05:17:10.889+0300 I REPL \u00a0\u00a0\u00a0\u00a0[rsSync] initial sync pending\r\n\r\n2018-08-28T05:17:12.709+0300 I REPL \u00a0\u00a0\u00a0\u00a0[ReplicationExecutor] syncing from: primaryserver\r\n\r\n2018-08-28T05:17:12.747+0300 I REPL \u00a0\u00a0\u00a0\u00a0[rsSync] initial sync drop all databases\r\n\r\n2018-08-28T05:17:12.747+0300 I STORAGE \u00a0[rsSync] dropAllDatabasesExceptLocal 4\r\n\r\n2018-08-28T05:17:16.312+0300 I REPL \u00a0\u00a0\u00a0\u00a0[rsSync] initial sync clone all databases\r\n\r\n2018-08-28T05:17:16.364+0300 I REPL \u00a0\u00a0\u00a0\u00a0[rsSync] fetching and creating collections for admin\r\n\r\n2018-08-28T05:17:16.374+0300 I REPL \u00a0\u00a0\u00a0\u00a0[rsSync] fetching and creating collections for db1\r\n\r\n2018-08-28T05:17:16.519+0300 I REPL \u00a0\u00a0\u00a0\u00a0[rsSync] fetching and creating collections for db2\r\n\r\n2018-08-28T05:17:16.532+0300 I REPL \u00a0\u00a0\u00a0\u00a0[rsSync] initial sync cloning db: admin<\/pre>\n
<\/p>\n
In this situation second method is more effective.<\/p>\n
You need to copy data files from an healthy member. Before copying you need to lock the healty secondary to prevent data files from changes.<\/p>\n
1- In healthy secondary run:<\/p>\n
db.fsyncLock()<\/pre>\n
<\/p>\n
2- shutdown the unhealthy mongo instance<\/p>\n
use admin\r\n\r\ndb.shutdownServer()<\/pre>\n
<\/p>\n
2- cp \/data<\/code> –> unhealthy server<\/p>\n
3-\u00a0<\/span>start the mongod instance<\/p>\n
4- unlock the healthy secondary:<\/p>\n
db.fsyncUnlock()<\/pre>\n
<\/p>\n
After copying data files replica member will syncronize itself from primary in a short time.<\/p>\n\n