I'm not quite sure what's going on, but it appears that our Lagotto installation isn't actually doing anything. I've tried queueing scrapes and then forcing sidekiq to start with:
RAILS_ENV=production bundle exec rake sidekiq:start
But nothing seems to change, our metrics are flatlined, am I not firing the correct command?
RAILS_ENV=production bundle exec rake sidekiq:start is the correct command, or
RAILS_ENV=production bundle exec rake sidekiq:monitor, the command that should run every hour via a cron job.
If nothing is happening, first make sure there are no sidekiq errors, e.g. in
log/sidekiq.log. Not all sidekiq errors appear in the Lagotto error reporting admin tab.
What do you see on the Status page? Do you have jobs under
I'm seeing similar issues. In fact, I was seeing the same issues with 3.18, which is why I gave up on it. I was hoping 3.20 fixed something.
Sidekiq is actually running, but not doing anything. I start a job to import articles, sometimes I get a blip on the dashboard, most times not, and nothing happens. I can't import works.
I sent you an email about this a few days ago Martin. I'm happy to give you access to the server that is playing up. I really need to get this fixed.
I take that back - it's doing something, but it has no articles to operate on, so it may be different from Andy's problem.
Hm. It might help if I take a closer look at both of your servers. We can do this via email.
no worries - send me your public key via email (email@example.com) and I'll add you to the list of authorized users and give you the IP.
Hey Martin - no word from you yet. Do you think you could respond with some suggestions? We're eager to upgrade from ALM to Lagotto.
Sorry, I am in Australia, returning tomorrow. Let's schedule a Skype call via email for next week.
very nice! whereabouts in Australia?
As discussed via Skype, the problem seems to be a bug in the CrossRef import script. This is fixed in the
3.21 released that I have just deployed.
Thanks, Martin. 3.21 appears to have fixed my import problem.
Afraid I'm still seeing this. Pretty much every source will report that Lagotto is attempting too many queries, then Sidekiq will stop doing any work and idle, even though there are jobs in the busy stack. It looks like a single source fails and instead of moving on, Lagotto waits for an hour before doing anything else...
We should talk about this over Skype. A good place to start is to look into the Sidekiq error log at
log/sidekiq.log to make sure there are no errors with the background workers.