2016-02-29 12:13:41.6546 (8605) [normal]: This program is published under the GNU General Public License, version 2
2016-02-29 12:13:41.6547 (8605) [normal]: For details see http://einstein.phys.uwm.edu/license.php
2016-02-29 12:13:41.6547 (8605) [normal]: This Einstein@home App was built at: Feb 19 2016 16:09:03
@herna: I'll also need the hostid of your host this snippet is from. Are you still seeing the task in the Manager? What does it say in the status column?
@Richard and @AgentB: this was the first thing that I noticed when I checked herna's file which was already reported by another user. I already know where to investigate and the two samples from AgentB and herna are a good help in that.
@herna: I'll also need the hostid of your host this snippet is from. Are you still seeing the task in the Manager? What does it say in the status column?
Einstein computer ID of the stuck upload task is 11711041. It's a Linux 3.13.0-37-generic (Mint) with Boinc 7.2.42 64-bit hosted as a virtual machine inside Windows 10 64-bit, most of the times with 2 Gb RAM and 7 or 8 cores. The guest Windows installation runs BRP6 ATI tasks, both run wuprop nci tasks alsongside.
The task is still shown as uploading, while _0 and _1 result files have uploaded, _2 returns the "length of file 0 bytes" error message.
I now know what must have happened but still not why. It seems that a small amount of tasks was assigned to two hosts at the same time. There are several checks on the server to prevent two work requests that are handled at the same time to issue the same tasks to two different hosts. It seems all of those checks failed to detect this case (for an unknown reason). I now added another check that should prevent such a case based on what we currently see in the logfiles (but still not the root cause). If this works can only be seen when the server is again under heavy load (hint: this will be the case later this week).
I'll shortly deploy a patch to the file upload handler so everyone who is affected by this can upload the surplus result files.
Snippet from client state.xml
)
Snippet from client state.xml - let me know if need any other logs/files, or when the upload can be aborted.
h1_0051.85_O1C01Cl1In1__O1AS20-100T_51.95Hz_374_1 44809.100000 44803.940719 0 4 x86_64-pc-linux-gnu 104 AVX2016-02-29 12:13:41.6546 (8605) [normal]: This program is published under the GNU General Public License, version 2
2016-02-29 12:13:41.6547 (8605) [normal]: For details see http://einstein.phys.uwm.edu/license.php
2016-02-29 12:13:41.6547 (8605) [normal]: This Einstein@home App was built at: Feb 19 2016 16:09:03
2016-02-29 12:13:41.6548 (8605) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O1AS20-100T_1.04_x86_64-pc-linux-gnu__AVX'.
2016-02-29 12:13:41.6729 (8605) [debug]: Successfully loaded libgcc_s.so.1
2016-02-29 12:13:41.6740 (8605) [debug]: BSGL output files
2016-02-29 12:13:41.6750 (8605) [debug]: Flags: LAL_NDEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, X64, SSE, SSE2, GNUC X86 GNUX86
2016-02-29 12:13:41.6751 (8605) [debug]: glibc version/release: 2.19/stable
2016-02-29 12:13:41.675160 - mytime()
2016-02-29 12:13:41.6755 (8605) [debug]: Set up communication with graphics process.
command line: ../../projects/einstein.phys.uwm.edu/einstein_O1AS20-100T_1.04_x86_64-pc-linux-gnu__AVX --refTime=1132729647.5 --Freq=51.95 --FreqBand=0.05 --dFreq=8.271845945e-07 --f1dot=-2.64248266531e-09 --f1dotBand=2.9e-09 --df1dot=1.3366502e-11 --gammaRefine=100 --computeBSGL --BSGLlogcorr=0 --Fstar0=65.826 --oLGX=0.001,0.001 --nCand1=10000 --SortToplist=6 --recalcToplistStats=1 -o ../../projects/einstein.phys.uwm.edu/h1_0051.85_O1C01Cl1In1__O1AS20-100T_51.95Hz_374_1_0 --printCand1 --semiCohToplist --ephemE=../../projects/einstein.phys.uwm.edu/earth00-19-DE405.dat --ephemS=../../projects/einstein.phys.uwm.edu/sun00-19-DE405.dat --segmentList=../../projects/einstein.phys.uwm.edu/O1AS20-100T.seg --FstatMethod=ResampBest --FstatMethodRecalc=DemodBest --numSkyPartitions=831 --partitionIndex=374 --gridType=3 --skyGridFile=../../projects/einstein.phys.uwm.edu/skygrid_GC_m0.001_0055Hz_O1AS20-100.dat --loudestSegOutput --getMaxFperSeg --peakThrF=2.6 --DataFiles1=../../projects/einstein.phys.uwm.edu/h1_0051.85_O1C01Cl1In1;../../projects/einstein.phys.uwm.edu/l1_0051.85_O1C01Cl1In1;../../projects/einstein.phys.uwm.edu/h1_0051.90_O1C01Cl1In1;../../projects/einstein.phys.uwm.edu/l1_0051.90_O1C01Cl1In1;../../projects/einstein.phys.uwm.edu/h1_0051.95_O1C01Cl1In1;../../projects/einstein.phys.uwm.edu/l1_0051.95_O1C01Cl1In1;../../projects/einstein.phys.uwm.edu/h1_0052.00_O1C01Cl1In1;../../projects/einstein.phys.uwm.edu/l1_0052.00_O1C01Cl1In1;../../projects/einstein.phys.uwm.edu/h1_0052.05_O1C01Cl1In1;../../projects/einstein.phys.uwm.edu/l1_0052.05_O1C01Cl1In1;../../projects/einstein.phys.uwm.edu/h1_0052.10_O1C01Cl1In1;../../projects/einstein.phys.uwm.edu/l1_0052.10_O1C01Cl1In1
Code-version: %% LAL: 6.15.2.1 (CLEAN 39e266a593dbd400f84a1f158d9513933d6a320c)
%% LALPulsar: 1.12.0.1 (CLEAN 39e266a593dbd400f84a1f158d9513933d6a320c)
%% LALApps: 6.17.1.1 (CLEAN 39e266a593dbd400f84a1f158d9513933d6a320c)
2016-02-29 12:13:42.5246 (8605) [normal]: Reading input data ... 2016-02-29 12:13:51.5337 (8605) [normal]: Search FstatMethod used: 'ResampGeneric'
2016-02-29 12:13:51.5338 (8605) [normal]: Recalc FstatMethod used: 'DemodSSE'
2016-02-29 12:13:52.4914 (8605) [normal]: Number of segments: 12, total number of SFTs in segments: 4744
done.
% --- GPS reference time = 1132729647.5000 , GPS data mid time = 1132729647.5000
2016-02-29 12:13:52.5046 (8605) [normal]: dFreqStack = 8.271846e-07, df1dot = 1.336650e-11, df2dot = 0.000000e+00, df3dot = 0.000000e+00
% --- Setup, N = 12, T = 755999 s, Tobs = 9044863 s, gammaRefine = 100, gamma2Refine = 484, gamma3Refine = 1
2016-02-29 12:13:52.6499 (8605) [normal]: INFO: No checkpoint checkpoint.cpt found - starting from scratch
% --- Cpt:0, total:25724, sky:1/118, f1dot:1/218
0.% --- CG:726252 FG:60447 f1dotmin_fg:-2.649099745508e-09 df1dot_fg:1.323416039604e-13 f2dotmin_fg:0 df2dot_fg:0 f3dotmin_fg:0 df3dot_fg:1
..............c
........................................................................................................................................c
...................................................................
1.......................................................................c
..........................................................................................................................................c
.........
2....................................................................................................................................c
......................................................................................
3......................................................c
.........................................................................................................................................c
...........................
4.............................................................................................................c
.............................................................................................................
5..............................c
...........................................................................................................................................c
.................................................
6.........................................................................................c
.................................................................................................................................
7.........c
.............................................................................................................................................c
....................................................................
8.......................................................................c
........................................................................................................................................c
...........
9...............................................................................................................................c
...........................................................................................
10...............................................c
............................................................................................................................................c
...............................
11.............................................................................................................c
.............................................................................................................
12.................................c
.........................................................................................................................................c
................................................
13.........................................................................................c
.................................................................................................................................
14........c
..........................................................................................................................................c
........................................................................
15................................................................c
.........................................................................................................................................c
.................
16.........................................................................................................................c
.................................................................................................
17..........................................c
..............................................................................................................................................c
..................................
18.........................................................................................................c
.................................................................................................................
19..........................c
............................................................................................................................................c
....................................................
20........................................................................................c
..................................................................................................................................
21.........c
..........................................................................................................................................c
.......................................................................
22........................................................................c
..............................................................................................................................................c
....
23........................................................................................................................................c
..................................................................................
24...........................................................c
..........................................................................................................................................c
.....................
25....................................................................................................................c
......................................................................................................
26.......................................c
............................................................................................................................................c
.......................................
27..................................................................................................c
........................................................................................................................
28................c
.........................................................................................................................................c
.................................................................
29.............................................................................c
..........................................................................................................................................c
...
30.........................................................................................................................................c
.................................................................................
31.........................................................c
..........................................................................................................................................c
.......................
32...................................................................................................................c
.......................................................................................................
33....................................c
..........................................................................................................................................c
............................................
34.................................................................................................c
.........................................................................................................................
35..................c
............................................................................................................................................c
............................................................
36...............................................................................c
...........................................................................................................................................
37...c
.......................................................................................................................................c
................................................................................
38..............................................................c
...........................................................................................................................................c
.................
39...........................................................................................................................c
...............................................................................................
40..........................................c
..............................................................................................................................................c
..................................
41..........................................................................................................c
................................................................................................................
42.............................c
..............................................................................................................................................c
...............................................
43..............................................................................................c
............................................................................................................................
44...............c
............................................................................................................................................c
...............................................................
45.........................................................................c
..........................................................................................................................................c
.......
46....................................................................................................................................c
......................................................................................
47.....................................................c
..........................................................................................................................................c
...........................
48.................................................................................................................c
.........................................................................................................
49...................................c
........................................................................................................................................c
...............................................
50...............................................................................................c
...........................................................................................................................
51.................c
..........................................................................................................................................c
...............................................................
52...........................................................................c
.......................................................................................................................................c
........
53..............................................................................................................................c
............................................................................................
54...........................................c
.............................................................................................................................................c
..................................
55............................................................................................................c
..............................................................................................................
56...............................c
.........................................................................................................................................c
..................................................
57.........................................................................................c
.................................................................................................................................
58............c
............................................................................................................................................c
..................................................................
59..........................................................................c
...........................................................................................................................................c
.....
60......................................................................................................................................c
....................................................................................
61...............................................c
............................................................................................................................c
...............................................
62...........................................................................................c
...............................................................................................................................
63...........c
......................................................................................................................................c
.........................................................................
64.................................................................c
........................................................................................................................................c
.................
65.........................................................................................................................c
.................................................................................................
66.........................................c
............................................................................................................................................c
.....................................
67....................................................................................................c
......................................................................................................................
68....................c
..........................................................................................................................................c
............................................................
69.............................................................................c
.............................................................................................................................................
70.c
.........................................................................................................................................c
................................................................................
71.........................................................c
.........................................................................................................................................c
........................
72......................................................................................................................c
....................................................................................................
73.........................................c
...............................................................................................................................................c
..................................
74..........................................................................................................c
................................................................................................................
75..........................c
...........................................................................................................................................c
.....................................................
76........................................................................................c
..................................................................................................................................
77...........c
............................................................................................................................................c
...................................................................
78........................................................................c
..........................................................................................................................................c
........
79.....................................................................................................................................c
.....................................................................................
80....................................................c
.............................................................................................................................................c
.........................
81....................................................................................................................c
......................................................................................................
82.................................c
..........................................................................................................................................c
...............................................
83..................................................................................................c
........................................................................................................................
84.....................c
..............................................................................................................................................c
.......................................................
85.........................................................c
...........................................................................................................................c
......................................
86...............................................................................................c
...........................................................................................................................
87...................c
.............................................................................................................................................c
..........................................................
88.....................................................................................c
.....................................................................................................................................
89.........c
................................................................................................................................c
.................................................................................
90.............................................................c
............................................................................................................................................c
.................
91.............................................................................................................................c
.............................................................................................
92..............................................c
................................................................................................................................................c
............................
93..................................................................................................................c
........................................................................................................
94.......................................c
...............................................................................................................................................c
....................................
95...........................................................................................................c
...............................................................................................................
96.............................c
................................................................................................................................................c
.............................................
97.................................................................................................c
.........................................................................................................................
98......................c
................................................................................................................................................c
....................................................
99..........................................................................................c
................................................................................................................................
100.................c
..............................................................................................................................................c
...........................................................
101..................................................................................c
........................................................................................................................................
102.....c
............................................................................................................................................c
.........................................................................
103....................................................................c
.............................................................................................................................................c
.........
104.....................................................................................................................................c
.....................................................................................
105.....................................................c
..............................................................................................................................................c
.......................
106........................................................................................................................c
..................................................................................................
107............................................c
..............................................................................................................................................c
................................
108..............................................................................................................c
............................................................................................................
109.................................c
...............................................................................................................................................c
..........................................
110..................................................................................................c
........................................................................................................................
111........................c
...........................................................................................................................................c
.......................................................
112......................................................................................c
....................................................................................................................................
113.....c
..............................................................................................................................................c
.......................................................................
114.......................................................................c
...........................................................................................................................................c
........
115.....................................................................................................................................c
.....................................................................................
116..........................................................c
.............................................................................................................................................c
...................
117.............................................................................................................................c
.............................................................................................
2016-03-01 20:16:35.0124 (8605) [normal]: Finished main analysis.
2016-03-01 20:16:35.0125 (8605) [normal]: Recalculating statistics for the final toplist...
2016-03-01 20:19:33.5792 (8605) [normal]: Finished recalculating toplist statistics.
2016-03-01 20:19:33.5793 (8605) [debug]: Writing output ... toplist2 ... toplist3 ... done.
2016-03-01 20:19:34.5159 (8605) [debug]: resultfile '../../projects/einstein.phys.uwm.edu/h1_0051.85_O1C01Cl1In1__O1AS20-100T_51.95Hz_374_1_0' (len 88), current config file: 0
2016-03-01 20:19:34.5160 (8605) [debug]: renaming '../../projects/einstein.phys.uwm.edu/h1_0051.85_O1C01Cl1In1__O1AS20-100T_51.95Hz_374_1_0-BSGLtL' to '../../projects/einstein.phys.uwm.edu/h1_0051.85_O1C01Cl1In1__O1AS20-100T_51.95Hz_374_1_1'
2016-03-01 20:19:34.5161 (8605) [debug]: renaming '../../projects/einstein.phys.uwm.edu/h1_0051.85_O1C01Cl1In1__O1AS20-100T_51.95Hz_374_1_0-BtSGLtL' to '../../projects/einstein.phys.uwm.edu/h1_0051.85_O1C01Cl1In1__O1AS20-100T_51.95Hz_374_1_2'
FPU status flags: COND_3 COND_1 COND_0 PRECISION
2016-03-01 20:19:34.5161 (8605) [normal]: done. calling boinc_finish(0).
20:19:34 (8605): called boinc_finish
]]>
@herna: I'll also need the
)
@herna: I'll also need the hostid of your host this snippet is from. Are you still seeing the task in the Manager? What does it say in the status column?
The updated BOINC web server
)
The updated BOINC web server software allows you to search by task name.
herna's task h1_0051.85_O1C01Cl1In1__O1AS20-100T_51.95Hz_374_1 leads to
Task 545989530
WU 239866864
Host 12088115
(which appears to belong to a different user)
RE: The updated BOINC web
)
which leads me to search and find task below
Strangeness on both these tasks shows more than the usual quorum members, plus at least one not being part of the quorum.
@Richard and @AgentB: this
)
@Richard and @AgentB: this was the first thing that I noticed when I checked herna's file which was already reported by another user. I already know where to investigate and the two samples from AgentB and herna are a good help in that.
I'll keep you posted how my investigation goes.
RE: @herna: I'll also need
)
Einstein computer ID of the stuck upload task is 11711041. It's a Linux 3.13.0-37-generic (Mint) with Boinc 7.2.42 64-bit hosted as a virtual machine inside Windows 10 64-bit, most of the times with 2 Gb RAM and 7 or 8 cores. The guest Windows installation runs BRP6 ATI tasks, both run wuprop nci tasks alsongside.
The task is still shown as uploading, while _0 and _1 result files have uploaded, _2 returns the "length of file 0 bytes" error message.
I now know what must have
)
I now know what must have happened but still not why. It seems that a small amount of tasks was assigned to two hosts at the same time. There are several checks on the server to prevent two work requests that are handled at the same time to issue the same tasks to two different hosts. It seems all of those checks failed to detect this case (for an unknown reason). I now added another check that should prevent such a case based on what we currently see in the logfiles (but still not the root cause). If this works can only be seen when the server is again under heavy load (hint: this will be the case later this week).
I'll shortly deploy a patch to the file upload handler so everyone who is affected by this can upload the surplus result files.
The file has uploaded, the
)
The file has uploaded, the task has reported and is no longer visible in the manager.
2717 Einstein@Home 07-03-2016 18:01 Started upload of h1_0051.85_O1C01Cl1In1__O1AS20-100T_51.95Hz_374_1_2
2718 Einstein@Home 07-03-2016 18:01 Finished upload of h1_0051.85_O1C01Cl1In1__O1AS20-100T_51.95Hz_374_1_2
2719 Einstein@Home 07-03-2016 18:01 Sending scheduler request: To report completed tasks.
2720 Einstein@Home 07-03-2016 18:01 Reporting 1 completed tasks
2721 Einstein@Home 07-03-2016 18:01 Not requesting tasks: "no new tasks" requested via Manager
2722 Einstein@Home 07-03-2016 18:01 Scheduler request completed
2723 Einstein@Home 07-03-2016 18:01 Completed result h1_0051.85_O1C01Cl1In1__O1AS20-100T_51.95Hz_374_1 refused: result already reported as success