Item10994: MongoDBPlugin can run out of indexes before they're enabled on _topic, _web and friends

Background:

• Sometimes imports don't run to completion. Maybe I interrupt a web that's only half-loaded. Could this be related to indexes not being set on the next run?
• Seems more reliable if I purge mongo completely before re-importing. But this doesn't guarantee a good import (somtimes end up with no indexes set, even from a clean mongodb)

Here's an example of what happened on an otherwise normal-looking import of Marine/SeaSlugs/Messages web:

> db.map.find({_id : "Marine/SeaSlugs/Messages"})
{ "hash" : "web_60f1085a95ad14944fee4a10fc8a1a34", "_id" : "Marine/SeaSlugs/Messages" }
> use web_60f1085a95ad14944fee4a10fc8a1a34
switched to db web_60f1085a95ad14944fee4a10fc8a1a34
> show collections
ACLProfiles
current
system.indexes
system.js
system.profile
> db.current.getIndexes()
[
   {
      "name" : "_id_",
      "ns" : "web_60f1085a95ad14944fee4a10fc8a1a34.current",
      "key" : {
         "_id" : 1
      },
      "v" : 0
   },
   {
      "ns" : "web_60f1085a95ad14944fee4a10fc8a1a34.current",
      "key" : {
         "FIELD.MessageID.value" : NumberLong(1)
      },
      "name" : "FIELD.MessageID",
      "background" : true,
      "v" : 0
   }
]
> 

Yes, I forgot to set MONITOR_INDEXES this time. Here is the output from that web, anyway (using -web all -recurse on -fork 1):

start web: Marine/SeaSlugs/Messages (36586) -> web_60f1085a95ad14944fee4a10fc8a1a34
imported 1000
imported 2000
imported 3000
imported 4000
imported 5000
imported 6000
imported 7000
imported 8000
imported 9000
imported 10000
imported 11000
imported 12000
imported 13000
imported 14000
imported 15000
imported 16000
imported 17000
imported 18000
imported 19000
imported 20000
imported 21000
imported 22000
imported 23000
imported 24000
imported 25000
imported 26000
imported 27000
imported 28000
imported 29000
imported 30000
imported 31000
imported 32000
imported 33000
imported 34000
imported 35000
imported 36000
imported 36587.14

Running again with monitor_index

-- PaulHarvey - 22 Jul 2011

considering that i actually call ensureIndex on every updateTopic call, because updates are much slower when there are no indexes, I really don't know - its basically bizzare.

-- SvenDowideit - 22 Jul 2011

Seems that CREATEINFO, TOPICINFO, _web and _topic aren't getting indexes set.

So I've updated https://gist.github.com/941875

var databases = db.adminCommand('listDatabases')['databases'];

for (var item in databases) {
    db = db.getMongo().getDB(item['name']);
    db.current.ensureIndex({_web: 1});
    db.current.ensureIndex({_topic: 1});
    db.current.ensureIndex({'TOPICINFO.author': 1})
    db.current.ensureIndex({'TOPICINFO.date': 1})
    db.current.ensureIndex({'CREATEINFO.author': 1})
    db.current.ensureIndex({'CREATEINFO.date': 1})
    db.current.ensureIndex({'TOPICPARENT.name': 1})
    db.current.ensureIndex({'FORM.name': 1})
}

-- PaulHarvey - 12 Aug 2011

Re-titled: this occurs when there are too many DataForm fields, we use up all the indexes on those first, before the important ones get set.

Failure mode of not having indexes set on _web or _topic etc. is that the default sorting will fail on large webs that can't be sorted in-memory by MongoDB.

-- PaulHarvey - 15 Mar 2012

Finally done, I think. Even added a test!

-- PaulHarvey - 16 Mar 2012

Re-opened to squelch the log noise down a bit (should only warn once per web-fieldname)

-- PaulHarvey - 26 Mar 2012

Commenting is disabled while not logged in