| Problem Reports |
|---|
| PR Number | Times | Time Lost | Description | Response |
|---|
| 4130 | 04:56-05:00 | 00:04 | Unceremonious abort, was not doing anything for 30 seconds prior to event, and before that was offsetting focus and sky. TT was closed. tcs_server was found to have crashed (when we checked it was not running, but it had been) after PR was issued, so TCS crashing may have caused the unceremonious abort. | |
| 4129 | 05:00-05:00 | 00:00 | The more humid it is outside the more negative the external temperature displays. (HET Weather Station GUI) | |
| 4131 | 05:55-06:03, 07:15 - 07:18, 07:24 - 07:27, 08:07 - 08:10, 08:37 - 08:40, 09:23 - 09:27, 09:33 - 09:37 | 00:28 | Velocity error: 32 (H1)
Following error: 22 (UX, LX, Rho)
Occurred after a period of notoffset_focus({ "dw": "-0.500000" }) doing anything, and then issuing dy_ang offsets and focus offsets. Last command issued that triggered fault was offset_focus({ "dw": "-0.500000" }) at (715, 78, 16, 0.19, 0.35, 3.06) ITF. TT was not closed at the time.
At 07:15 (-07:18)
Velocity error: 32 (H1)
All other error codes read 0.
After commanding offset_sky({ "dx_ang": "0", "dy_ang": "60" }) 4 times in rapid succession and commanding offset_focus({ "dw": "-0.500000" }) 2 times in rapid succession got an H1 velocity fault. After fault tracker was around the same ITF coords as first instance of PR, when velocity faults occurred.
At 07:24
Velocity error: 32 (H1)
All other error codes read 0.
After commanding offset_sky({ "dx_ang": "0", "dy_ang": "60" }) 1 time in rapid succession and commanding offset_focus({ "dw": "-0.500000" }) 4 times in rapid succession got an H1 velocity fault. TT was NOT in closed loop. After fault tracker was at ITF (1417, -6, 69, 0.43, -0.03, 6.08)
At 08:07:
Velocity error: 32 (H1)
All other error codes read 0.
After commanding offset_sky({ "dx_ang": "0", "dy_ang": "60" }) 1 time in rapid succession and commanding offset_focus({ "dw": "-0.500000" }) 2 times in rapid succession got an H1 velocity fault. TTC was NOT in closed loop mode. After fault tracker was at ITF (1314, 28, 64, 0.42, 0.12, 5.63)
At 08:37:
Velocity error: 32 (H1)
All other error codes read 0.
After commanding offset_sky({ "dx_ang": "60", "dy_ang": "0" }) 1 time in rapid succession and commanding offset_focus({ "dw": "-0.500000" }) 2 times in rapid succession got an H1 velocity fault. TTC was NOT in closed loop mode. When fault occurred tracker was at ITF (-479, 46, 8, -0.16, 0.21, -2.04)
At 09:23 (cycle 73-- sorry, should have been writing these from the start)
Velocity error: 32 (H1)
All other error codes read 0.
After commanding offset_focus({ "dw": "-0.500000" }) 3 times in rapid succession and commanding offset_sky({ "dx_ang": "60", "dy_ang": "0" }) 1 time in rapid succession got an H1 velocity fault. TTC was NOT in closed loop mode. When fault occurred tracker was at ITF (-917, 73, 23, -0.31, 0.32, -3.93)
At the end of cycle 78 (09:33) we got a HFC/Master fault after:
offset_sky({ "dx_ang": "0", "dy_ang": "-60" })
offset_sky({ "dx_ang": "0", "dy_ang": "-60" })
offset_focus({ "dw": "-0.500000" })
offset_focus({ "dw": "-0.500000" })
offset_sky({ "dx_ang": "0", "dy_ang": "60" })
We couldn't tell if any Velocity Error Code appeared. At time of fault tracker was at ITF (-934, 68, 28, -0.32, 0.30, -4.0) | |
| 4132 | 06:14-06:18, 08:39 - 08:42, 08:49 - 08:51 | 00:09 | Attempting to load a trajectory caused tcs_server to use 100% of CPU.
At 08:39 after an abort (while slewing to start of track) attempting to load a trajectory caused tcs_server to use 300% of CPU.
At 08:49 after an abort (while slewing to start of track) attempting to load a trajectory caused tcs_server to use 100% of CPU.
If a "failed to acquire trajectory" message occurs it seems like tcs successfully exits the routine with an error message. Otherwise it seems like it might be getting caught in an infinite loop somewhere. | |
| 4133 | 06:25-06:27 | 00:02 | Upon bringing up tcs gui Motion lights on Tracker tab all appeared gray. Motion lights remained gray even when tracker was commanded to move from TCS, and moved. Issuing an abort command produced the following error:
tracker_abort caught an exception: 'Timeout waiting for message ID 2 ACK. Receiver for tcsGuiTcsAsyncCommandSender is likely offline.' | |
| 4136 | 08:16-08:19 | 00:03 | go_next() command was not working; RA/Dec were not updating. These problems indicated that the tcs server crashed. | Shutdown everything and brought it back up. |
| 4137 | 05:00-05:00 | 00:00 | After HFC/Master Fault, next go_next() command seems to consistently track to start of track instead of slewing which of course messes up everything. (Or at least forces us to abort and try to load_trajectory()/go_next() again, which usually works.) This happened consistently tonight, and we were usually a good distance from start of next track. (0.4+ hours in RA away from current RA; Dec remained consistent throughout testing tonight.) | |
| 4134 | 08:02-08:04, 09:31 - 09:33, 09:50 - 09:52 | 00:06 | Tcs Command Error resulted in aborting track, produced following error message:
{
"__exception_type": 0,
"__exception_string": "remote exception: MOVA_RTF() TMCS reports failure for command index 12474: MOVA_RTF 12474 C_ERROR 17262 nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
",
"__exception_file": "unknown",
"__exception_line": 0,
"__exception_func": "unknown",
"__effective_id": 183,
"__original_id": 183,
"__origin": "tcsGuiTcsAsyncCommandSender.jove.23243.0175af122->tcp://192.168.66.31:30300",
"__handler": "offset_sky",
"__async": "true",
"__done": "true",
"__ack": "false",
"__error": "true",
"__data_time": "1442908798.592892718",
"__wire_time": "1442908798.592894115",
"__data": "false",
"__pid": 7963
}
Occurred again at 09:31, at the end of cycle 76.
Occurred again at 09:50 at the end of cycle 81.
After the last two occurrences (not sure about first one) issuing go_next() command caused tcs_server to use 100% of cpu and hang. | |
| 4135 | 06:44-06:48, 06:56-07:00, 07:06-07:10, 07:32-07:36, 07:44-07:48, 08:25-08:29, 08:33-08:37, 09:13-09:17 | 00:32 | At 06:44 HFC/Master Fault triggered with no error codes displaying. After issuing offset_focus({ "dw": "-0.500000" }) 3 times.
06:56: HFC/Master Fault triggered with no error codes displaying. After issuing offset_focus({ "dw": "-0.500000" }) 2 times in rapid succession (TO suspects focus offsets applied too quickly; just after last focus command turned green in 'command status' box on the TCS GUI) (TT was in closed loop mode)
07:06: HFC/Master Fault triggered with no error codes displaying. After issuing offset_focus({ "dw": "-0.500000" }) 5 times in rapid succession. It looks like this is a way to consistently trigger HFC/Master Fault without error codes. (TT was NOT in closed loop mode.)
07:32: HFC/Master Fault triggered with no error codes displaying. After issuing offset_sky({ "dx_ang": "0", "dy_ang": "60" }) one time and offset_focus({ "dw": "-0.500000" }) 3 times in rapid succession. This combination was reliably causing the H1 velocity fault, but either it did not this time or we did not see the error code before it auto-cleared. After fault tracker was at ITF (1100, -5, 37.5, 0.34, -0.02, 4.71)
07:44: HFC/Master Fault triggered with no error codes displaying. After issuing offset_sky({ "dx_ang": "0", "dy_ang": "60" }) one time and offset_focus({ "dw": "-0.500000" }) 2 times in rapid succession. After fault tracker was at ITF (1046, 19, 35, 0.33, 0.08, 4.48)
08:25: HFC/Master Fault triggered with no error codes displaying. After issuing offset_sky({ "dx_ang": "0", "dy_ang": "-60" }) three times and offset_focus({ "dw": "-0.500000" }) 3 times in rapid succession. After fault tracker was at ITF (1137, 3, 48, 0.36, 0.01, 4.88)
08:33: HFC/Master Fault triggered with no error codes displaying. After issuing offset_sky({ "dx_ang": "0", "dy_ang": "-60" }) three times and offset_focus({ "dw": "-0.500000" }) 2 times in rapid succession. After fault tracker was at ITF (138, -12, -1.7, -0.04, -0.04, -0.59)
09:13: HFC/Master Fault triggered with no error codes displaying. After issuing offset_sky({ "dx_ang": "0", "dy_ang": "-60" }) three times and offset_focus({ "dw": "-0.500000" }) 3 times in rapid succession. After fault tracker was at ITF (-910, 60, 26, -0.3, 0.28, -3.89). It seemed much more difficult to get the tracker to misbehave at this position, but then again its only one data point among many.
(Upon further testing around X=-910 it did seem more stable.) | |