rr recording gdb (in batch mode) fails, attaching as well #3964
Description
rr gdb --batch -ex start -x $HOME/r0_x86_rdrand.gdbserver-ret.gdb --args cobcrun PROG ARG
rr: Saving execution to trace directory `/home/so/.local/share/rr/gdb-2'.
Temporary breakpoint 1 at 0x400e80: file ../../bin/cobcrun.c, line 391.
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
Temporary breakpoint 1, main (argc=3, argv=0x7fffffffbfe8) at ../../bin/cobcrun.c:391
391 setlocale (LC_ALL, "");
Function "r0_x86_rdrand" not defined.
Breakpoint 2 (r0_x86_rdrand) pending.
Breakpoint 3 (PROG_:SECTION_SOME) pending.
[New Thread 0x7ffff7eb06c0 (LWP 1885453)]
Thread 2 "cobcrun" received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 0x7ffff7eb06c0 (LWP 1885453)]
0x00007ffff6cabe65 in clone () from /usr/lib64/libc.so.6
[FATAL src/Scheduler.cc:358:is_task_runnable() errno: EDOM]
(task 1884222 (rec:1884222) at time 1574273)
-> Assertion `t->emulated_stop_type == GROUP_STOP' failed to hold.
Tail of trace dump:
{
real_time:13410002.279985 global_time:1574253, event:`SYSCALL: select' (state:ENTERING_SYSCALL) tid:1884212, ticks:751654953
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x7ffe183fb9d0 rdi:0xd rbp:0x0 rsp:0x681ffd40 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0x0 r14:0x7ffe183fb9d0 r15:0x681fffa0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x17 fs_base:0x7f1fa5823c80 gs_base:0x0
}
{
real_time:13410002.280059 global_time:1574254, event:`SYSCALL: select' (state:EXITING_SYSCALL) tid:1884212, ticks:751654953
rax:0x1 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x7ffe183fb9d0 rdi:0xd rbp:0x0 rsp:0x681ffd40 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0x0 r14:0x7ffe183fb9d0 r15:0x681fffa0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x17 fs_base:0x7f1fa5823c80 gs_base:0x0
{ tid:1884212, addr:0x7ffe183fb9d0, length:0x8 }
}
{
real_time:13410002.280100 global_time:1574255, event:`SYSCALLBUF_FLUSH' tid:1884212, ticks:751655086
{ syscall:'read', ret:0x0, size:0x10, desched:1 }
}
{
real_time:13410002.280102 global_time:1574256, event:`SYSCALL: rt_sigaction' (state:ENTERING_SYSCALL) tid:1884212, ticks:751655086
rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x7ffe183fb780 rsi:0x7ffe183fb6e0 rdi:0x14 rbp:0x7ffe183fba80 rsp:0x7ffe183fb6e0 r8:0x7ffe183fb8d0 r9:0x7f1fa318d860 r10:0x8 r11:0x246 r12:0x7ffe183fb9d0 r13:0x0 r14:0xd r15:0xc rip:0x7f1fa2e1ac24 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xd fs_base:0x7f1fa5823c80 gs_base:0x0
}
{
real_time:13410002.280103 global_time:1574257, event:`SYSCALLBUF_RESET' tid:1884212, ticks:751655086
}
{
real_time:13410002.280126 global_time:1574258, event:`SYSCALL: rt_sigaction' (state:EXITING_SYSCALL) tid:1884212, ticks:751655086
rax:0x0 rbx:0x0 rcx:0xffffffffffffffff rdx:0x7ffe183fb780 rsi:0x7ffe183fb6e0 rdi:0x14 rbp:0x7ffe183fba80 rsp:0x7ffe183fb6e0 r8:0x7ffe183fb8d0 r9:0x7f1fa318d860 r10:0x8 r11:0x246 r12:0x7ffe183fb9d0 r13:0x0 r14:0xd r15:0xc rip:0x7f1fa2e1ac24 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xd fs_base:0x7f1fa5823c80 gs_base:0x0
{ tid:1884212, addr:0x7ffe183fb780, length:0x20 }
}
{
real_time:13410002.280189 global_time:1574259, event:`SYSCALLBUF_FLUSH' tid:1884212, ticks:751656068
{ syscall:'close', ret:0x0, size:0x10 }
{ syscall:'openat', ret:0xfffffffffffffffe, size:0x10, desched:1 }
}
{
real_time:13410002.280192 global_time:1574260, event:`SYSCALL: rt_sigaction' (state:ENTERING_SYSCALL) tid:1884212, ticks:751656068
rax:0xffffffffffffffda rbx:0x5f0690 rcx:0xffffffffffffffff rdx:0x7ffe183fb800 rsi:0x7ffe183fb760 rdi:0x2 rbp:0xa9cc20 rsp:0x7ffe183fb760 r8:0x7ffe183fb950 r9:0x5a38 r10:0x8 r11:0x246 r12:0x7ffe183fbab0 r13:0x0 r14:0xf3f0c0 r15:0x0 rip:0x7f1fa2e1ac24 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xd fs_base:0x7f1fa5823c80 gs_base:0x0
}
{
real_time:13410002.280193 global_time:1574261, event:`SYSCALLBUF_RESET' tid:1884212, ticks:751656068
}
{
real_time:13410002.280214 global_time:1574262, event:`SYSCALL: rt_sigaction' (state:EXITING_SYSCALL) tid:1884212, ticks:751656068
rax:0x0 rbx:0x5f0690 rcx:0xffffffffffffffff rdx:0x7ffe183fb800 rsi:0x7ffe183fb760 rdi:0x2 rbp:0xa9cc20 rsp:0x7ffe183fb760 r8:0x7ffe183fb950 r9:0x5a38 r10:0x8 r11:0x246 r12:0x7ffe183fbab0 r13:0x0 r14:0xf3f0c0 r15:0x0 rip:0x7f1fa2e1ac24 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xd fs_base:0x7f1fa5823c80 gs_base:0x0
{ tid:1884212, addr:0x7ffe183fb800, length:0x20 }
}
{
real_time:13410002.280257 global_time:1574263, event:`SYSCALL: tkill' (state:ENTERING_SYSCALL) tid:1884212, ticks:751657499
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x1 rsi:0x9 rdi:0x1cc50d rbp:0x1cc50d rsp:0x681ffd40 r8:0x1772018 r9:0x1cc50d r10:0x19f4960 r11:0x246 r12:0x9 r13:0x6aa6a0 r14:0x0 r15:0x681fffa0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc8 fs_base:0x7f1fa5823c80 gs_base:0x0
}
{
real_time:13410002.280285 global_time:1574264, event:`SYSCALL: tkill' (state:EXITING_SYSCALL) tid:1884212, ticks:751657499
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x1 rsi:0x9 rdi:0x1cc50d rbp:0x1cc50d rsp:0x681ffd40 r8:0x1772018 r9:0x1cc50d r10:0x19f4960 r11:0x246 r12:0x9 r13:0x6aa6a0 r14:0x0 r15:0x681fffa0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc8 fs_base:0x7f1fa5823c80 gs_base:0x0
}
{
real_time:13410002.280317 global_time:1574265, event:`SYSCALL: ptrace' (state:ENTERING_SYSCALL) tid:1884212, ticks:751657523
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1cc50d rdi:0x8 rbp:0x7f1fa58236c8 rsp:0x681ffd40 r8:0x7 r9:0x1cc50d r10:0x0 r11:0x246 r12:0x1cc03e r13:0x6aa6a0 r14:0x0 r15:0x681fffa0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x65 fs_base:0x7f1fa5823c80 gs_base:0x0
}
{
real_time:13410002.280335 global_time:1574266, event:`SYSCALL: ptrace' (state:EXITING_SYSCALL) tid:1884212, ticks:751657523
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1cc50d rdi:0x8 rbp:0x7f1fa58236c8 rsp:0x681ffd40 r8:0x7 r9:0x1cc50d r10:0x0 r11:0x246 r12:0x1cc03e r13:0x6aa6a0 r14:0x0 r15:0x681fffa0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x65 fs_base:0x7f1fa5823c80 gs_base:0x0
}
{
real_time:13410002.280366 global_time:1574267, event:`SYSCALL: tkill' (state:ENTERING_SYSCALL) tid:1884212, ticks:751657553
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x9 rdi:0x1cc03e rbp:0x1cc03e rsp:0x681ffd40 r8:0x1cc50d r9:0x1cc03e r10:0x7 r11:0x246 r12:0x9 r13:0x6aa6a0 r14:0x0 r15:0x681fffa0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc8 fs_base:0x7f1fa5823c80 gs_base:0x0
}
{
real_time:13410002.281080 global_time:1574268, event:`SYSCALL: tkill' (state:EXITING_SYSCALL) tid:1884212, ticks:751657553
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x9 rdi:0x1cc03e rbp:0x1cc03e rsp:0x681ffd40 r8:0x1cc50d r9:0x1cc03e r10:0x7 r11:0x246 r12:0x9 r13:0x6aa6a0 r14:0x0 r15:0x681fffa0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc8 fs_base:0x7f1fa5823c80 gs_base:0x0
}
{
real_time:13410002.281121 global_time:1574269, event:`SYSCALL: ptrace' (state:ENTERING_SYSCALL) tid:1884212, ticks:751657577
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1cc03e rdi:0x8 rbp:0x7f1fa58236c8 rsp:0x681ffd40 r8:0x7 r9:0x1cc03e r10:0x0 r11:0x246 r12:0x1cc03e r13:0x6aa6a0 r14:0x0 r15:0x681fffa0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x65 fs_base:0x7f1fa5823c80 gs_base:0x0
}
{
real_time:13410002.281140 global_time:1574270, event:`SYSCALL: ptrace' (state:EXITING_SYSCALL) tid:1884212, ticks:751657577
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1cc03e rdi:0x8 rbp:0x7f1fa58236c8 rsp:0x681ffd40 r8:0x7 r9:0x1cc03e r10:0x0 r11:0x246 r12:0x1cc03e r13:0x6aa6a0 r14:0x0 r15:0x681fffa0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x65 fs_base:0x7f1fa5823c80 gs_base:0x0
}
{
real_time:13410002.281176 global_time:1574271, event:`SYSCALL: wait4' (state:ENTERING_SYSCALL) tid:1884212, ticks:751657614
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x40000000 rsi:0x0 rdi:0x1cc50d rbp:0x0 rsp:0x681ffd40 r8:0x0 r9:0x1cc03e r10:0x0 r11:0x246 r12:0x40000000 r13:0x1cc50d r14:0x0 r15:0x681fffa0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3d fs_base:0x7f1fa5823c80 gs_base:0x0
}
{
real_time:13410002.281239 global_time:1574272, event:`EXIT' tid:1885453, ticks:0
{ tid:1885453, addr:0x7ffff7eb0990, length:0x4 }
}
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackERNS_8ScopedFdE+0x29)[0x469739]
rr(_ZN2rr15emergency_debugEPNS_4TaskE+0x11c)[0x5742cc]
rr[0x56405c]
rr[0x5641c0]
rr[0x5641f9]
rr(_ZN2rr9Scheduler16is_task_runnableEPNS_10RecordTaskERNS_14WaitAggregatorEPb+0x18a)[0x4bbbea]
rr(_ZN2rr9Scheduler23find_next_runnable_taskERNS_14WaitAggregatorERSt3mapIiSt6vectorIPNS_10RecordTaskESaIS6_EESt4lessIiESaISt4pairIKiS8_EEEPbi+0xb5)[0x4bc905]
rr(_ZN2rr9Scheduler10rescheduleENS_10SwitchableE+0x510)[0x4bd500]
rr(_ZN2rr13RecordSession11record_stepEv+0xbd)[0x54463d]
rr(_ZN2rr13RecordCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0xb09)[0x54e689]
rr(main+0x157)[0x44e1d7]
/usr/lib64/libc.so.6(__libc_start_main+0xe5)[0x7ffb2b5c3d85]
rr(_start+0x2e)[0x44e3be]
=== End rr backtrace
Launch debugger with
'gdb' '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:49214' '/opt/gnucobol/bin/cobcrun'
[FATAL src/Task.cc:1180:regs() errno: EDOM]
(task 1884212 (rec:1884212) at time 1574273)
-> Assertion `stopped_or_unexpected_exit()' failed to hold.
[FATAL src/TraceStream.cc:1515:close() errno: EBADF] Unable to write /home/so/.local/share/rr/gdb-2/incomplete
=== Start rr backtrace:
rr[0x565430]
rr[0x44f95d]
rr[0x4334be]
rr[0x563e5f]
rr[0x5641c0]
rr[0x5641f9]
rr(_ZNK2rr4Task4regsEv+0x2f)[0x4915af]
rr[0x586340]
rr(_ZN2rr9GdbServer25dispatch_debugger_requestERNS_7SessionERKNS_10GdbRequestENS0_11ReportStateE+0x1977)[0x591f67]
rr(_ZN2rr9GdbServer25process_debugger_requestsENS0_11ReportStateE+0xbd)[0x59440d]
rr(_ZN2rr15emergency_debugEPNS_4TaskE+0x2a9)[0x574459]
rr[0x56405c]
rr[0x5641c0]
rr[0x5641f9]
rr(_ZN2rr9Scheduler16is_task_runnableEPNS_10RecordTaskERNS_14WaitAggregatorEPb+0x18a)[0x4bbbea]
rr(_ZN2rr9Scheduler23find_next_runnable_taskERNS_14WaitAggregatorERSt3mapIiSt6vectorIPNS_10RecordTaskESaIS6_EESt4lessIiESaISt4pairIKiS8_EEEPbi+0xb5)[0x4bc905]
rr(_ZN2rr9Scheduler10rescheduleENS_10SwitchableE+0x510)[0x4bd500]
rr(_ZN2rr13RecordSession11record_stepEv+0xbd)[0x54463d]
rr(_ZN2rr13RecordCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0xb09)[0x54e689]
rr(main+0x157)[0x44e1d7]
/usr/lib64/libc.so.6(__libc_start_main+0xe5)[0x7ffb2b5c3d85]
rr(_start+0x2e)[0x44e3be]
=== End rr backtrace
Aborted (core dumped)
As expected rr waited until gdb-server was used for attaching, but then aborted directly...
This did work two years ago #3577 (comment) (also with the much faster option to record gdbserver, then execute those commands with a "remote" gdb - but it seems that option now leads to gdbserver waiting forever while gdb is in "continue mode" and wait that something happens after it gets one time into "clone()" in a second thread and I manually execute continue in there)
The script has
set breakpoint pending on
break r0_x86_rdrand
commands
silent
return
continue
end
b PROG_:SECTION_SOME
commands
#silent
p some_var = 535
continue
end
# difference from starting under GDB - gdbserver already started the program, so don't run
continue