Recovering from power outage

So, it’s summertime and I’m in the middle of a desert. That means it gets hot, air conditioners are running and of course, power outages happen. That’s what I’m going through now, at least. It doesn’t help that they outlast my UPS and systems shut down.

I’m trying to restart a job that had been running for 30+ hours, yeah, it’s a pretty large dataset by any measure. I’m not exactly sure where in the scheme of things it was as far as processing goes. I’m hoping that by posting the log text from where it was when the power went out, someone may recognize it and be able to say, “Oh, that output is from Georeferencing, you should start over there.” That’s my hope anyway.

Here’s the output from the job it was running when I lost power:
2020-07-22 04:32:43,541 INFO: -------------------------------------------------------
2020-07-22 04:32:44,127 INFO: DJI_0047_h4hois7.JPG resection inliers: 4024 / 4026
2020-07-22 04:32:44,601 DEBUG: Ceres Solver Report: Iterations: 5, Initial cost: 4.583134e+02, Final cost: 9.120773e+01, Termination: CONVERGENCE
2020-07-22 04:32:44,605 INFO: Adding DJI_0047_h4hois7.JPG to the reconstruction
2020-07-22 04:32:51,542 INFO: -------------------------------------------------------
2020-07-22 04:32:52,083 INFO: DJI_0067_e76Ge05.JPG resection inliers: 4078 / 4081
2020-07-22 04:32:52,576 DEBUG: Ceres Solver Report: Iterations: 5, Initial cost: 1.948527e+02, Final cost: 1.158514e+02, Termination: CONVERGENCE
2020-07-22 04:32:52,579 INFO: Adding DJI_0067_e76Ge05.JPG to the reconstruction
2020-07-22 04:32:53,830 INFO: Re-triangulating
2020-07-22 04:32:53,839 INFO: Shots and/or GCPs are well-conditionned. Using naive 3D-3D alignment.
2020-07-22 04:34:05,161 INFO: Shots and/or GCPs are well-conditionned. Using naive 3D-3D alignment.
2020-07-22 04:39:15,253 DEBUG: Ceres Solver Report: Iterations: 19, Initial cost: 6.893106e+03, Final cost: 5.520602e+03, Termination: CONVERGENCE

As I’m looking at the “Restart” menu, I can’t help but to think that it would be great if it showed what jobs were actually completed and what job was processing when it was interrupted. So, here’s my feature request that is spun from this issue. Could some kind of progress tracking be put in the interface that will show what jobs completed? Here’s a mockup of what comes to mind.

image

2 Likes

Quick trick, you can simply restart from the last step (Report). If other steps haven’t been processed, they will be processed first.

We should probably add a “Resume” button to make this more evident.

4 Likes

Awesome! I did not know about that. I’ll use that in the future because with this heat, I see this problem happening more.

2 Likes