Results 1 to 3 of 3
  1. #1
    Join Date
    Jul 2015
    Posts
    2

    Unanswered: mysqld crashes on start with or without innodb_force_recovery

    Hello!

    This is from a tiny DB that only has a couple of tables and probably hasn't been written to (only read from) in years. It's running on a debian wheezy vserver, that had to be reverted to a 1 day old backup. After that, mysqld refuses to start even with innodb_force_recovery set to 6. I don't exactly know, why I'm doing a "innodb recovery", because there definitely was no data loss - relevant files for the tables have been unchanged for weeks at least.

    The last time I had any trouble with mysql, "innodb" hasn't even been around, so I'm utterly lost as to what's that about now. And i'm not even entirely sure if "innodb" was already part of mysql the last time that server was rebooted.

    The configuration files have never been touched by anything other than apt-get. I tried all settings from innodb_force_recovery=0 to 6, hoping to make it "just shut up and give me back my database that is obviously still there, still fine, probably still exactly the same as a year ago", but I did not manage to get mysqld running at all. This is what the syslog looks like:

    Code:
    Jul  6 21:04:43 host mysqld_safe: Starting mysqld daemon with databases from /mysql
    Jul  6 21:04:43 host mysqld: 150706 21:04:43 [Note] /usr/sbin/mysqld (mysqld 5.5.43-0+deb7u1-log) starting as process 11403 ...
    Jul  6 21:04:43 host mysqld: 150706 21:04:43 [Note] Plugin 'FEDERATED' is disabled.
    Jul  6 21:04:43 host mysqld: 150706 21:04:43 InnoDB: The InnoDB memory heap is disabled
    Jul  6 21:04:43 host mysqld: 150706 21:04:43 InnoDB: Mutexes and rw_locks use GCC atomic builtins
    Jul  6 21:04:43 host mysqld: 150706 21:04:43 InnoDB: Compressed tables use zlib 1.2.7
    Jul  6 21:04:43 host mysqld: 150706 21:04:43 InnoDB: Using Linux native AIO
    Jul  6 21:04:43 host mysqld: 150706 21:04:43 InnoDB: Initializing buffer pool, size = 128.0M
    Jul  6 21:04:43 host mysqld: 150706 21:04:43 InnoDB: Completed initialization of buffer pool
    Jul  6 21:04:43 host mysqld: 150706 21:04:43 InnoDB: highest supported file format is Barracuda.
    Jul  6 21:04:43 host mysqld: InnoDB: The log sequence number in ibdata files does not match
    Jul  6 21:04:43 host mysqld: InnoDB: the log sequence number in the ib_logfiles!
    Jul  6 21:04:43 host mysqld: 150706 21:04:43  InnoDB: Database was not shut down normally!
    Jul  6 21:04:43 host mysqld: InnoDB: Starting crash recovery.
    Jul  6 21:04:43 host mysqld: InnoDB: Reading tablespace information from the .ibd files...
    Jul  6 21:04:43 host mysqld: InnoDB: Restoring possible half-written data pages from the doublewrite
    Jul  6 21:04:43 host mysqld: InnoDB: buffer...
    Jul  6 21:04:43 host mysqld: InnoDB: Warning: a page in the doublewrite buffer is not within space
    Jul  6 21:04:43 host mysqld: InnoDB: bounds; space id 0 page number 901, page 1 in doublewrite buf.
    Jul  6 21:04:43 host mysqld: InnoDB: Warning: a page in the doublewrite buffer is not within space
    Jul  6 21:04:43 host mysqld: InnoDB: bounds; space id 0 page number 902, page 2 in doublewrite buf.
    Jul  6 21:04:43 host mysqld: InnoDB: Warning: a page in the doublewrite buffer is not within space
    Jul  6 21:04:43 host mysqld: InnoDB: bounds; space id 0 page number 903, page 3 in doublewrite buf.
    Jul  6 21:04:43 host mysqld: InnoDB: Warning: a page in the doublewrite buffer is not within space
    Jul  6 21:04:43 host mysqld: InnoDB: bounds; space id 0 page number 904, page 4 in doublewrite buf.
    Jul  6 21:04:43 host mysqld: InnoDB: Warning: a page in the doublewrite buffer is not within space
    Jul  6 21:04:43 host mysqld: InnoDB: bounds; space id 0 page number 896, page 124 in doublewrite buf.
    Jul  6 21:04:43 host mysqld: InnoDB: Warning: a page in the doublewrite buffer is not within space
    Jul  6 21:04:43 host mysqld: InnoDB: bounds; space id 0 page number 897, page 125 in doublewrite buf.
    Jul  6 21:04:43 host mysqld: InnoDB: Warning: a page in the doublewrite buffer is not within space
    Jul  6 21:04:43 host mysqld: InnoDB: bounds; space id 0 page number 898, page 126 in doublewrite buf.
    Jul  6 21:04:43 host mysqld: InnoDB: Warning: a page in the doublewrite buffer is not within space
    Jul  6 21:04:43 host mysqld: InnoDB: bounds; space id 0 page number 899, page 127 in doublewrite buf.
    Jul  6 21:04:43 host mysqld: InnoDB: Error: trying to access page number 0 in space 0,
    Jul  6 21:04:43 host mysqld: InnoDB: space name ./ibdata1,
    Jul  6 21:04:43 host mysqld: InnoDB: which is outside the tablespace bounds.
    Jul  6 21:04:43 host mysqld: InnoDB: Byte offset 0, len 16384, i/o type 10.
    Jul  6 21:04:43 host mysqld: InnoDB: If you get this error at mysqld startup, please check that
    Jul  6 21:04:43 host mysqld: InnoDB: your my.cnf matches the ibdata files that you have in the
    Jul  6 21:04:43 host mysqld: InnoDB: MySQL server.
    Jul  6 21:04:43 host mysqld: 150706 21:04:43  InnoDB: Assertion failure in thread 3064453344 in file fil0fil.c line 4578
    Jul  6 21:04:43 host mysqld: InnoDB: We intentionally generate a memory trap.
    Jul  6 21:04:43 host mysqld: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
    Jul  6 21:04:43 host mysqld: InnoDB: If you get repeated assertion failures or crashes, even
    Jul  6 21:04:43 host mysqld: InnoDB: immediately after the mysqld startup, there may be
    Jul  6 21:04:43 host mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to
    Jul  6 21:04:43 host mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
    Jul  6 21:04:43 host mysqld: InnoDB: about forcing recovery.
    Jul  6 21:04:43 host mysqld: 19:04:43 UTC - mysqld got signal 6 ;
    Jul  6 21:04:43 host mysqld: This could be because you hit a bug. It is also possible that this binary
    Jul  6 21:04:43 host mysqld: or one of the libraries it was linked against is corrupt, improperly built,
    Jul  6 21:04:43 host mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
    Jul  6 21:04:43 host mysqld: We will try our best to scrape up some info that will hopefully help
    Jul  6 21:04:43 host mysqld: diagnose the problem, but since we have already crashed, 
    Jul  6 21:04:43 host mysqld: something is definitely wrong and this may fail.
    Jul  6 21:04:43 host mysqld: 
    Jul  6 21:04:43 host mysqld: key_buffer_size=16777216
    Jul  6 21:04:43 host mysqld: read_buffer_size=131072
    Jul  6 21:04:43 host mysqld: max_used_connections=0
    Jul  6 21:04:43 host mysqld: max_threads=151
    Jul  6 21:04:43 host mysqld: thread_count=0
    Jul  6 21:04:43 host mysqld: connection_count=0
    Jul  6 21:04:43 host mysqld: It is possible that mysqld could use up to 
    Jul  6 21:04:43 host mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346075 K  bytes of memory
    Jul  6 21:04:43 host mysqld: Hope that's ok; if not, decrease some variables in the equation.
    Jul  6 21:04:43 host mysqld: 
    Jul  6 21:04:43 host mysqld: Thread pointer: 0x0
    Jul  6 21:04:43 host mysqld: Attempting backtrace. You can use the following information to find out
    Jul  6 21:04:43 host mysqld: where mysqld died. If you see no messages after this, something went
    Jul  6 21:04:43 host mysqld: terribly wrong...
    Jul  6 21:04:43 host mysqld: stack_bottom = 0 thread_stack 0x30000
    Jul  6 21:04:43 host mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x33)[0xb725e2d3]
    Jul  6 21:04:43 host mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0xb70fef44]
    Jul  6 21:04:43 host mysqld: [0xb6db2500]
    Jul  6 21:04:43 host mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
    Jul  6 21:04:43 host mysqld: information that should help you find out what is causing the crash.
    Jul  6 21:04:43 host mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
    Jul  6 21:04:57 host /etc/init.d/mysql[11600]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
    Jul  6 21:04:57 host /etc/init.d/mysql[11600]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed
    Jul  6 21:04:57 host /etc/init.d/mysql[11600]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
    Jul  6 21:04:57 host /etc/init.d/mysql[11600]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
    Jul  6 21:04:57 host /etc/init.d/mysql[11600]:
    No idea what to try next (besides restoring an even older backup)... so any hints are appreciated!

  2. #2
    Join Date
    Jun 2011
    Posts
    31
    Provided Answers: 1
    Take a backup of your crashed tables and edit my.cnf file and add

    Innodb_force_recovery = 1
    It helps to find corrupted tables. Basically it can re-build one or two corrupted tables.

    1. Stop the mysqld.
    2. Backup /var/lib/mysql/ib*

    Add these lines into my.cnf

    1. innodb_force_recovery = 1
    2. Restart mysqld.
    3. Dump all tables:# mysqldump -A > dump.sql
    4. Now drop all the databases which need recovery.
    5. Stop mysqld.
    6. Remove /var/lib/mysql/ib*
    7. Comment out innodb_force_recovery in my.cnf
    8. Restart mysqld. Look at mysql error log. By default it should be /var/lib/mysql/server/host_name.com.err to see the creation of new ib* files.
    9. Restore databases from the dump:mysql < dump.sql​

  3. #3
    Join Date
    Jul 2015
    Posts
    2

    ...

    I already tried "Innodb_force_recovery = 1" (and 0..6 - see OP, I think I wrote it twice ). Wasn't able to try the rest, because mysqld still wouldn't start - at all.

    After a while ( Would have updated this thread sooner, but it was invisible for some reason. :/ ), I got tired of innodb and simply deleted /mysql/ib*. Although it complained about something ( "blah, blah, maybe you forgot to delete frm files or something" ), everything worked fine after that, no data corruption, everything still there. So... while I would have preferred a more graceful solution, that's "fixed" now.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •