Results 1 to 4 of 4
  1. #1
    Join Date
    Feb 2009
    Posts
    11

    Unanswered: Out of Memory issue

    Hello,

    I was hoping to get some assistance with a memory problem on a new mysql server.


    System info:
    Windows 2008 Standard
    in VMware ESX 3.5 environment
    CPU: Quad core
    RAM allocated: 6 GB (will have 18GB allocated later)
    64-bit system
    Mysql version: community 5.1.34
    innoDB database


    We have migrated some of our database from a 32-bit Win2003 system running MySQL community version 5.0 to this 64-bit system Windows system. We used mySQL Administrator backup and Restore. So we have changed the version of MySQL, the OS and to 64-bit...


    We have allocated 6 GB to the server in a VMWare environment. The previous 32-bit system was ALSO in a VMware environment. We did not experience this problem on the 32-bit windows 2003 system.


    When I perform a query on a large table using the MySQL Query Browser:
    select * from archivetable
    (archivetable has 11,000,000 rows and has 9.9 GB data, 1.3GB index)


    I get the following error popup:
    Glib-ERROR **:gmem.c:173: Failed to allocate 29280000 bytes aborting...

    and the Query browser closes.

    Also, Memory usage on the system ramps up to about 80% until the program closes.
    If i use the MySQL command line the memory usage incrases to 99% usage and no results are returned!


    Obviously this is a memory issue but I am not sure what settings to change in my.ini


    Any advice is welcome.

    Show Variables:

    Variable_name = Value
    auto_increment_increment = 1
    auto_increment_offset = 1
    autocommit = ON
    automatic_sp_privileges = ON
    back_log = 50
    basedir = C:\Program Files\MySQL\MySQL Server 5.1\
    big_tables = OFF
    binlog_cache_size = 32768
    binlog_format = STATEMENT
    bulk_insert_buffer_size = 8388608
    character_set_client = utf8
    character_set_connection = utf8
    character_set_database = latin1
    character_set_filesystem = binary
    character_set_results = utf8
    character_set_server = latin1
    character_set_system = utf8
    character_sets_dir = C:\Program Files\MySQL\MySQL Server 5.1\share\charsets\
    collation_connection = utf8_general_ci
    collation_database = latin1_swedish_ci
    collation_server = latin1_swedish_ci
    completion_type = 0
    concurrent_insert = 1
    connect_timeout = 10
    datadir = D:\MySQL\MySQL Server 5.1\Data\
    date_format = %Y-%m-%d
    datetime_format = %Y-%m-%d %H:%i:%s
    default_week_format = 0
    delay_key_write = ON
    delayed_insert_limit = 100
    delayed_insert_timeout = 300
    delayed_queue_size = 1000
    div_precision_increment = 4
    engine_condition_pushdown = ON
    error_count = 0
    event_scheduler = OFF
    expire_logs_days = 0
    flush = OFF
    flush_time = 1800
    foreign_key_checks = ON
    ft_boolean_syntax = + -><()~*:&|
    ft_max_word_len = 84
    ft_min_word_len = 4
    ft_query_expansion_limit = 20
    ft_stopword_file = (built-in)
    general_log = OFF
    general_log_file = D:\MySQL\MySQL Server 5.1\Data\OurDatabase.log
    group_concat_max_len = 1024
    have_community_features = YES
    have_compress = YES
    have_crypt = NO
    have_csv = YES
    have_dynamic_loading = YES
    have_geometry = YES
    have_innodb = YES
    have_ndbcluster = NO
    have_openssl = DISABLED
    have_partitioning = YES
    have_query_cache = YES
    have_rtree_keys = YES
    have_ssl = DISABLED
    have_symlink = YES
    hostname = OurDatabase
    identity = 0
    ignore_builtin_innodb = OFF
    init_connect =
    init_file =
    init_slave =
    innodb_adaptive_hash_index = ON
    innodb_additional_mem_pool_size = 33554432
    innodb_autoextend_increment = 8
    innodb_autoinc_lock_mode = 1
    innodb_buffer_pool_size = 4194304000
    innodb_checksums = ON
    innodb_commit_concurrency = 0
    innodb_concurrency_tickets = 500
    innodb_data_file_path = ibdata1:10M:autoextend
    innodb_data_home_dir = D:\MySQL\MySQL Server 5.1\data\
    innodb_doublewrite = ON
    innodb_fast_shutdown = 1
    innodb_file_io_threads = 4
    innodb_file_per_table = ON
    innodb_flush_log_at_trx_commit = 2
    innodb_flush_method =
    innodb_force_recovery = 0
    innodb_lock_wait_timeout = 50
    innodb_locks_unsafe_for_binlog = OFF
    innodb_log_buffer_size = 16777216
    innodb_log_file_size = 426770432
    innodb_log_files_in_group = 2
    innodb_log_group_home_dir = .\
    innodb_max_dirty_pages_pct = 90
    innodb_max_purge_lag = 0
    innodb_mirrored_log_groups = 1
    innodb_open_files = 300
    innodb_rollback_on_timeout = OFF
    innodb_stats_on_metadata = ON
    innodb_support_xa = ON
    innodb_sync_spin_loops = 20
    innodb_table_locks = ON
    innodb_thread_concurrency = 8
    innodb_thread_sleep_delay = 10000
    insert_id = 0
    interactive_timeout = 28800
    join_buffer_size = 131072
    keep_files_on_create = OFF
    key_buffer_size = 57671680
    key_cache_age_threshold = 300
    key_cache_block_size = 1024
    key_cache_division_limit = 100
    language = C:\Program Files\MySQL\MySQL Server 5.1\share\english\
    large_files_support = ON
    large_page_size = 0
    large_pages = OFF
    last_insert_id = 0
    lc_time_names = en_US
    license = GPL
    local_infile = ON
    log = OFF
    log_bin = OFF
    log_bin_trust_function_creators = OFF
    log_bin_trust_routine_creators = OFF
    log_error = D:\MySQL\MySQL Server 5.1\Data\OurDatabase.err
    log_output = FILE
    log_queries_not_using_indexes = OFF
    log_slave_updates = OFF
    log_slow_queries = ON
    log_warnings = 1
    long_query_time = 2.000000
    low_priority_updates = OFF
    lower_case_file_system = ON
    lower_case_table_names = 1
    max_allowed_packet = 1048576
    max_binlog_cache_size = 4294963200
    max_binlog_size = 1073741824
    max_connect_errors = 10
    max_connections = 800
    max_delayed_threads = 20
    max_error_count = 64
    max_heap_table_size = 16777216
    max_insert_delayed_threads = 20
    max_join_size = 18446744073709551615
    max_length_for_sort_data = 1024
    max_prepared_stmt_count = 16382
    max_relay_log_size = 0
    max_seeks_for_key = 4294967295
    max_sort_length = 1024
    max_sp_recursion_depth = 0
    max_tmp_tables = 32
    max_user_connections = 0
    max_write_lock_count = 4294967295
    min_examined_row_limit = 0
    multi_range_count = 256
    myisam_data_pointer_size = 6
    myisam_max_sort_file_size = 107374182400
    myisam_recover_options = OFF
    myisam_repair_threads = 1
    myisam_sort_buffer_size = 131072000
    myisam_stats_method = nulls_unequal
    myisam_use_mmap = OFF
    named_pipe = OFF
    net_buffer_length = 16384
    net_read_timeout = 30
    net_retry_count = 10
    net_write_timeout = 60
    new = OFF
    old = OFF
    old_alter_table = OFF
    old_passwords = OFF
    open_files_limit = 2048
    optimizer_prune_level = 1
    optimizer_search_depth = 62
    optimizer_switch = index_merge=on,index_merge_union=on,index_merge_so rt_union=on,index_merge_intersection=on
    pid_file = D:\MySQL\MySQL Server 5.1\Data\OurDatabase.pid
    plugin_dir = C:\Program Files\MySQL\MySQL Server 5.1\lib/plugin
    port = 3306
    preload_buffer_size = 32768
    profiling = OFF
    profiling_history_size = 15
    protocol_version = 10
    pseudo_thread_id = 6
    query_alloc_block_size = 8192
    query_cache_limit = 1048576
    query_cache_min_res_unit = 4096
    query_cache_size = 367001600
    query_cache_type = ON
    query_cache_wlock_invalidate = OFF
    query_prealloc_size = 8192
    rand_seed1 =
    rand_seed2 =
    range_alloc_block_size = 4096
    read_buffer_size = 65536
    read_only = OFF
    read_rnd_buffer_size = 262144
    relay_log =
    relay_log_index =
    relay_log_info_file = relay-log.info
    relay_log_purge = ON
    relay_log_space_limit = 0
    report_host =
    report_password =
    report_port = 3306
    report_user =
    rpl_recovery_rank = 0
    secure_auth = OFF
    secure_file_priv =
    server_id = 0
    shared_memory = OFF
    shared_memory_base_name = MYSQL
    skip_external_locking = ON
    skip_networking = OFF
    skip_show_database = OFF
    slave_compressed_protocol = OFF
    slave_exec_mode = STRICT
    slave_load_tmpdir = C:\Windows\TEMP
    slave_net_timeout = 3600
    slave_skip_errors =
    slave_transaction_retries = 10
    slow_launch_time = 2
    slow_query_log = ON
    slow_query_log_file =
    sort_buffer_size = 262144
    sql_auto_is_null = ON
    sql_big_selects = ON
    sql_big_tables = OFF
    sql_buffer_result = OFF
    sql_log_bin = ON
    sql_log_off = OFF
    sql_log_update = ON
    sql_low_priority_updates = OFF
    sql_max_join_size = 18446744073709551615
    sql_mode = STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_ SUBSTITUTION
    sql_notes = ON
    sql_quote_show_create = ON
    sql_safe_updates = OFF
    sql_select_limit = 18446744073709551615
    sql_slave_skip_counter =
    sql_warnings = OFF
    ssl_ca =
    ssl_capath =
    ssl_cert =
    ssl_cipher =
    ssl_key =
    storage_engine = InnoDB
    sync_binlog = 0
    sync_frm = ON
    system_time_zone = AUS Eastern Standard Time
    table_definition_cache = 256
    table_lock_wait_timeout = 50
    table_open_cache = 619
    table_type = InnoDB
    thread_cache_size = 38
    thread_handling = one-thread-per-connection
    thread_stack = 262144
    time_format = %H:%i:%s
    time_zone = SYSTEM
    timed_mutexes = OFF
    timestamp = 1244083897
    tmp_table_size = 131072000
    tmpdir = C:\Windows\TEMP
    transaction_alloc_block_size = 8192
    transaction_prealloc_size = 4096
    tx_isolation = REPEATABLE-READ
    unique_checks = ON
    updatable_views_with_limit = YES
    version = 5.1.34-community-log
    version_comment = MySQL Community Server (GPL)
    version_compile_machine = unknown
    version_compile_os = Win64
    wait_timeout = 28800
    warning_count = 0


    thanks for any assistance!

  2. #2
    Join Date
    Feb 2009
    Posts
    11
    ok, seems this is a MySQL browser feature that halts a query that is too large.
    I have never seen this occur on our 32-bit system. perhaps becuase the system would never allocate all of the memory to the query...or perhaps because I never let the query run long enough.

    so, assuming this is not a "problem" then the real problem may be that MySQL queries can consume all of the physical memory on the system.

    when the memory is consumed I notice that the windows system is unresponsive.
    Is there a way to stop that all of the physcial memory being used by MySQL processes/queries?

    sorry that I am so vague but I have never had this happen before, for one reason or another.

    thanks for any assistance.

  3. #3
    Join Date
    Jun 2007
    Location
    London
    Posts
    2,527
    Quote Originally Posted by DJmysql
    select * from archivetable
    (archivetable has 11,000,000 rows and has 9.9 GB data, 1.3GB index)
    Is this really an issue? Does your application actually need to pull 11m rows from your database at one time?

  4. #4
    Join Date
    Feb 2009
    Posts
    11
    Quote Originally Posted by mike_bike_kite
    Is this really an issue? Does your application actually need to pull 11m rows from your database at one time?
    hey, thanks for the reply.
    you are absolutely correct- I do not think it is an issue anymore (see below) and really, we will not be pulling 11million rows in a real-life query.

    it was simply my misunderstanding about some basics, i think, that generated my enquiry.

    i ran some tests with MySQL Query Browser and the mysql command line, together with different innodb_buffer settings and different amounts of allocated RAM to the VM , and from that gained more of an understanding about the way memory is/may be handled.

    i still have some questions about RAM/paging etc. in the context of MySQL but may run some more tests firsts.

    i will also run some "real-life" web queries against the DB and see how it performs.

    thanks again!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •