Hmm.  Bit of a stumbling block this.

I have been playing as follows; refreshing 'adventure counters' - the little red, yellow, blue, red gems' - ONLY on the locations which sport a 'gem' surrounded by a sun-ray halo.  You know the ones.  So far, so good.  Now, here's the question.  I have been refreshing these immediately after a challenge is completed...  now, although the following is not stated in the rules, I'm starting to suspect that these sunburst locations should only be refreshed after an event card is drawn. 

There's some text in the rulebook about, and I paraphrase, 'these sunburst icons on the event cards remind players to refresh adventure/location counters'.  Now, either I'm not naturally forgetful and remember, as a matter of course, to replace the gems on the sunburst locations, or I'm playing this game incorrectly and should only be refreshing said gems when an event card is played.  There is no clarification to be found in the rules.

Can anyone advise?

Thanks in advance.

You should only refresh adventure counters when Event card is drawn.

Thanks GOING DOWN.  Thought that must be the case, but was uncertain as wording in Rules is very uncler.