分类 MySQL 下的文章

CentOS7.x快速安装Mariadb10.4

MariaDB数据库管理系统是MySQL的一个分支,主要由开源社区在维护,采用GPL授权许可 MariaDB的目的是完全兼容MySQL,包括API和命令行,使之能轻松成为MySQL的代替品。在存储引擎方面,使用XtraDB(英语:XtraDB)来代替MySQL的InnoDB。 MariaDB由MySQL的创始人Michael Widenius(英语:Michael Widenius)主导开发,他早前曾以10亿美元的价格,将自己创建的公司MySQL AB卖给了SUN,此后,随着SUN被甲骨文收购,MySQL的所有权也落入Oracle的手中。MariaDB名称来自Michael Widenius的女儿Maria的名字。

mariadb.png
CentOS7快速安装Mariadb10.04方法如下:
1、检查是否已经具有MariaDB相关安装,并删除已有安装:

rpm -qa|grep mariadb  #查询已安装包
rpm -e --nodeps mariadb-*   #移出相关包
yum remove mysql mysql-server mysql-libs compat-mysql51   # 删除Mysql服务

2、增加MariaDB的仓库源,使用官方脚本会自动安装最新仓库:

yum install wget -y
wget https://downloads.mariadb.com/MariaDB/mariadb_repo_setup
chmod +x mariadb_repo_setup
./mariadb_repo_setup
yum install MariaDB-server -y
systemctl start mariadb
systemctl enable mariadb

3、初始化安全设置:

mysql_secure_installation   #根据提示操作修改密码等


---阅读剩余部分---

MySQL触发器的创建、查看、删除

介绍
表述:顾名思义在条件符合某种情况时,被触发执行的相关操作。
时间:事件发生前before和发生后after
事件:

名称    介绍
update    可用**old**访问旧数据,**new**访问新数据
delete    可用**old**访问旧数据
insert    可用**new**访问新数据 

创建触发器:

delimiter //  重定义行结束符
    create trigger trigger_name trigger_time trigger_event on table_name for each row
    begin
    -- 需要执行的sql操作语句
    end//
delimiter ;
-- delimiter          用于修改默认的行结束符,由于在触发器中有多条sql语句,他们需要使用分号来结束,但是触发器是一个整体,所以需要先更换默认的行结束符
-- trigger_name     自定义触发器名
-- trigger_time        时间点 before|after
-- trigger_event     事件 delete|update|insert
-- table_name         表名



---阅读剩余部分---

ERROR 1786 (HY000): Statement violates GTID consistency: CREATE TABLE ... SELECT.问题解决

ogp-mysql57-gtid-replication-b.png
MySQL使用Create table select的时候报错如下:

root@monitor-db 19:35:25 [test]>create table a3 select * from a2 ;
ERROR 1786 (HY000): Statement violates GTID consistency: CREATE TABLE ... SELECT.

问题分析:
MySQL开启gtid以后就不能使用了,MySQL只允许能够保障事务安全,并且能够被日志记录的SQL语句被执行,像create table … select 和 create temporarytable语句,以及同时更新事务表和非事务表的SQL语句或事务都不允许执行

root@monitor-db 19:35:12 [test]>show global variables like '%gtid_mode%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| gtid_mode     | ON    |
+---------------+-------+
1 row in set (0.00 sec)

如业务需要,可以在线关闭掉gtid,执行如下SQL关闭gtid:

set global gtid_mode='ON_PERMISSIVE';

set global gtid_mode='OFF_PERMISSIVE';

SET @@GLOBAL.ENFORCE_GTID_CONSISTENCY = OFF;

set global gtid_mode='OFF';

show global variables like '%gtid_mode%';

查询执行结果:

root@monitor-db 19:21:31 [test]>show global variables like '%gtid_mode%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| gtid_mode     | OFF   |
+---------------+-------+
1 row in set (0.00 sec)

再次创建:

root@monitor-db 19:32:46 [test]>desc a2;
+-----------------+------------------+------+-----+---------+-------+
| Field           | Type             | Null | Key | Default | Extra |
+-----------------+------------------+------+-----+---------+-------+
| id              | int(10) unsigned | NO   |     | 0       |       |
| title           | varchar(100)     | NO   |     | NULL    |       |
| author          | varchar(40)      | NO   |     | NULL    |       |
| submission_date | date             | YES  |     | NULL    |       |
+-----------------+------------------+------+-----+---------+-------+
4 rows in set (0.00 sec)

root@monitor-db 19:32:51 [test]>create table a3 select * from a2 ;
Query OK, 0 rows affected (0.01 sec)
Records: 0  Duplicates: 0  Warnings: 0

root@monitor-db 19:32:57 [test]>desc a3;
+-----------------+------------------+------+-----+---------+-------+
| Field           | Type             | Null | Key | Default | Extra |
+-----------------+------------------+------+-----+---------+-------+
| id              | int(10) unsigned | NO   |     | 0       |       |
| title           | varchar(100)     | NO   |     | NULL    |       |
| author          | varchar(40)      | NO   |     | NULL    |       |
| submission_date | date             | YES  |     | NULL    |       |
+-----------------+------------------+------+-----+---------+-------+
4 rows in set (0.01 sec)

已经OK;
在线打开GTID:

root@monitor-db 19:41:31 [test]>set global gtid_mode='OFF_PERMISSIVE';
Query OK, 0 rows affected (0.00 sec)

root@monitor-db 19:44:14 [test]>
root@monitor-db 19:44:14 [test]>set global gtid_mode='ON_PERMISSIVE';
Query OK, 0 rows affected (0.00 sec)

root@monitor-db 19:44:14 [test]>
root@monitor-db 19:44:14 [test]>SET @@GLOBAL.ENFORCE_GTID_CONSISTENCY = ON;
Query OK, 0 rows affected (0.00 sec)

root@monitor-db 19:44:14 [test]>
root@monitor-db 19:44:14 [test]>set global gtid_mode='ON';
Query OK, 0 rows affected (0.01 sec)

root@monitor-db 19:44:14 [test]>
root@monitor-db 19:44:14 [test]>show global variables like '%gtid_mode%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| gtid_mode     | ON    |
+---------------+-------+
1 row in set (0.00 sec)

打开ok,MySQL5.7开启/关闭GTID,不用重启服务很方便;

MySQL网络超时错误Got timeout reading communication packets的解决方法

MySQL大量报错Got timeout reading communication packets,error显示日志如下:

2018-11-14T23:27:01.130760Z 2876756 [Note] Aborted connection 2876756 to db: 'zabbix_db' user: 'zabbix_u' host: '192.168.255.209' (Got timeout reading communication packets)
2018-11-14T23:27:03.437908Z 2874970 [Note] Aborted connection 2874970 to db: 'zabbix_db' user: 'zabbix_u' host: '192.168.255.209' (Got timeout reading communication packets)

当前MySQL主要timeout相关的参数如下:

show session variables like '%timeout%';
+-----------------------------+----------+
| Variable_name               | Value    |
+-----------------------------+----------+
| connect_timeout             | 10       |
| delayed_insert_timeout      | 300      |
| have_statement_timeout      | YES      |
| innodb_flush_log_at_timeout | 1        |
| innodb_lock_wait_timeout    | 10       |
| innodb_rollback_on_timeout  | ON       |
| interactive_timeout         | 60       |
| lock_wait_timeout           | 31536000 |
| net_read_timeout            | 60      |
| net_write_timeout           | 30      |
| rpl_stop_slave_timeout      | 31536000 |
| slave_net_timeout           | 60       |
| wait_timeout                | 900      |
+-----------------------------+----------+
13 rows in set (0.01 sec)

connect_timeout:默认为10S
wait_timeout:默认是8小时,即28800秒
interactive_timeout:默认是8小时,即28800秒
net_read_timeout:默认是30S
net_write_timeout:默认是60S
如果是认证过程中超时,不管是读还是,都是 connect_timeout;
对于读网络超时,一般是 wait_timeout/interactive_timeout,基本不会是 net_read_timeout(特例是业务用到 LOAD DATA LOCAL FILE);
对于写网络超时,都是 net_write_timeout。
解决方法如下:

set global interactive_timeout=900;
set global net_read_timeout=900;
set global net_write_timeout=900;

再次tail日志,已经无该信息报错了,可以把以上参数加入到my.cnf里面,900根据实际场景设置;

参考文档:
http://mysql.taobao.org/monthly/2017/05/04/
http://cenalulu.github.io/mysql/mysql-timeout/
https://www.cnblogs.com/xiaoboluo768/p/6222862.html

MySQL使用存储过程插入100万条记录

业务上为了测试DB写入性能,需测试大量数据写入速度,数据库部署在aws上,4核8G内存100G硬盘,MySQL5.7.27
mysql.png

--创建测试数据库
create database a1 default charset utf8mb4;
--创建测试表
CREATE TABLE `test110` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `name` varchar(200) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `user` varchar(200) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `comp` varchar(200) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `create_time` datetime DEFAULT NULL,
  `cid` int(11) DEFAULT NULL,
  `aid` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
--创建插入存储过程
DELIMITER $$
CREATE PROCEDURE test_insert()
BEGIN
declare i int;
set i=0;
while i<1000000 do
INSERT INTO `test110` VALUES (null,'TMP01','user11','会员中台','2019-08-15 00:00:02','11','22');
set i=i+1;
end while;
end $$
DELIMITER ;

测试如下,速度很快:

mysql> call test_insert();
Query OK, 1 row affected (44.71 sec)

mysql> call test_insert();
Query OK, 1 row affected (44.74 sec)

mysql> call test_insert();
Query OK, 1 row affected (44.71 sec)

mysql> select count(1) from test110;
+----------+
| count(1) |
+----------+
|  3000000 |
+----------+
1 row in set (0.44 sec)

MySQL配置如下my.cnf

[client]
default-character-set = utf8mb4
port        = 3306
socket        = /data/mysql/mysql.sock

[mysql]
default-character-set = utf8mb4
prompt="\u@oatestdb \R:\m:\s [\d]> "
no-auto-rehash

[mysqld]

secure_file_priv = ''
#validate-password = off
slow_query_log = On
log_output = FILE
slow-query-log-file=/data/mysql/slow.log
long_query_time = 30 
log_queries_not_using_indexes = ON

lower_case_table_names=1
auto-increment-increment = 1
auto-increment-offset = 1

log_warnings=1
log_error_verbosity=1

default-storage-engine=INNODB
group_concat_max_len = 18446744073709551615
# generic configuration options
port        = 3306
socket        = /data/mysql/mysql.sock 
pid-file        = /data/mysql/mysqld.pid
basedir        = /usr/local/mysql
datadir         = /data/mysql/db
language    = /usr/local/mysql/share/english 
user        = mysql
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
skip-external-locking
skip-name-resolve
skip-character-set-client-handshake
character-set-server = utf8mb4
explicit_defaults_for_timestamp = true
init_connect='SET NAMES utf8mb4'
character-set-client-handshake = FALSE
collation-server = utf8mb4_unicode_ci
explicit_defaults_for_timestamp

#replication
expire_logs_days = 30
sync-binlog = 1000
server_id=1
gtid_mode=ON
enforce_gtid_consistency=ON
master_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE
log_slave_updates=ON
log_bin=/data/mysql/binlog/binlog
binlog_format=ROW


relay_log_recovery = 1
skip_slave_start = 0
slave-skip-errors = all
binlog_cache_size = 4M
max_binlog_cache_size = 2G
max_binlog_size = 1G

#group replication
transaction_write_set_extraction=XXHASH64
loose-group_replication_group_name="e4668cea-d7ca-11e6-86b5-18a99b76310d"
loose-group_replication_start_on_boot=off
loose-group_replication_local_address= "db1:24901"
loose-group_replication_group_seeds= "db1:24901,db2:24902,db3:24903"
loose-group_replication_bootstrap_group= off
loose-group_replication_single_primary_mode=FALSE
loose-group_replication_enforce_update_everywhere_checks= TRUE

skip_name_resolve = 1
open_files_limit    = 65535
back_log = 1024
max_connections = 5000
max_connect_errors = 100000
max_prepared_stmt_count=1048576
table_open_cache = 10240
table_definition_cache = 10240
table_open_cache_instances = 64
thread_stack = 512K
external-locking = FALSE
max_allowed_packet = 32M
sort_buffer_size = 16M
join_buffer_size = 16M
thread_cache_size = 7680
query_cache_size = 0
query_cache_type = 0
interactive_timeout = 900
wait_timeout = 900
tmp_table_size = 256M
max_heap_table_size = 256M

# innodb
innodb_buffer_pool_size = 12G 
innodb_data_file_path = IBdata1:1024M;IBdata2:1024M:autoextend
innodb_temp_data_file_path = ibtmp1:12M:autoextend:max:5G
innodb_thread_concurrency = 16
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 32M
innodb_log_file_size = 2G
innodb_log_files_in_group = 3
innodb_max_dirty_pages_pct = 90
innodb_flush_method=O_DSYNC

innodb_io_capacity = 50000
innodb_io_capacity_max = 80000
innodb_flush_neighbors = 0
innodb_write_io_threads = 8
innodb_read_io_threads = 8
innodb_purge_threads = 4
innodb_page_cleaners = 4
innodb_open_files = 65535
innodb_max_dirty_pages_pct = 50
innodb_flush_method = O_DIRECT
innodb_lru_scan_depth = 4000
innodb_checksum_algorithm = crc32
#innodb_file_format = Barracuda
#innodb_file_format_max = Barracuda
innodb_lock_wait_timeout = 10
innodb_rollback_on_timeout = 1
innodb_print_all_deadlocks = 1
innodb_file_per_table = 1
innodb_online_alter_log_max_size = 4G
internal_tmp_disk_storage_engine = InnoDB
innodb_stats_on_metadata = 0

[mysqld_safe]
open-files-limit = 8192
log-error=/data/mysql/error.log
pid-file=/data/mysql/mysqld.pid

[mysqldump]
quick
max_allowed_packet = 32M

MySQL查询所有视图并给用户授予指定视图权限

视图其实就是一张虚拟的表,所有也可以认为是一张表,MySQL查询所有视图:

show table status where comment='view'\G;

查询视图结构:

 DESC master_data_user;
+-----------+--------------+------+-----+---------+-------+
| Field     | Type         | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+-------+
| username  | varchar(255) | YES  |     | NULL    |       |
| org_id    | varchar(255) | YES  |     | NULL    |       |
| status    | varchar(255) | YES  |     | NULL    |       |
| name      | varchar(255) | YES  |     | NULL    |       |
| org_name  | varchar(255) | YES  |     | NULL    |       |
| email     | varchar(255) | YES  |     | NULL    |       |
| telephone | varchar(255) | YES  |     | NULL    |       |
| post      | varchar(255) | YES  |     | NULL    |       |
+-----------+--------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

或者

SHOW TABLE STATUS LIKE 'master_data_user'\G;
*************************** 1. row ***************************
           Name: master_data_user
         Engine: NULL
        Version: NULL
     Row_format: NULL
           Rows: NULL
 Avg_row_length: NULL
    Data_length: NULL
Max_data_length: NULL
   Index_length: NULL
      Data_free: NULL
 Auto_increment: NULL
    Create_time: NULL
    Update_time: NULL
     Check_time: NULL
      Collation: NULL
       Checksum: NULL
 Create_options: NULL
        Comment: VIEW
1 row in set (0.00 sec)

创建data_view用户并授予单个视图权限:

grant select,show view on escdb.master_data_org to data_view@'192.168.%' IDENTIFIED BY 'xxxxxxx';
grant select,show view on escdb.master_data_user to data_view@'192.168.%' ;
flush privileges;

data_view登录后即只能查看master_data_org和master_data_user视图;
查看当前登录用户权限:

 show grants;
+----------------------------------------------------------------------------------+
| Grants for data_view@192.168.%                                                   |
+----------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'data_view'@'192.168.%'                                    |
| GRANT SELECT, SHOW VIEW ON `escdb`.`master_data_org` TO 'data_view'@'192.168.%'  |
| GRANT SELECT, SHOW VIEW ON `escdb`.`master_data_user` TO 'data_view'@'192.168.%' |
+----------------------------------------------------------------------------------+
3 rows in set (0.00 sec)

MySQL子查询参数说明

MySQL查询的五种子句

where子句(条件查询):按照“条件表达式”指定的条件进行查询。

group by子句(分组):按照“属性名”指定的字段进行分组。group by子句通常和count()、sum()等聚合函数一起使用。

having子句(筛选):有group by才能having子句,只有满足“条件表达式”中指定的条件的才能够输出。

order by子句(排序):按照“属性名”指定的字段进行排序。排序方式由“asc”和“desc”两个参数指出,默认是按照“asc”来排序,即升序。

limit(限制结果集)。

---阅读剩余部分---

MySQL视图查看及权限说明

查看视图是指查看数据库中,已经存在的视图的定义 必须要有SHOW VIEW的权限
查看视图有三种方式:
DESCRIBE语句
使用DESCRIBE语句,查看视图
MySQL中,使用DESCRIBE可以查看视图的字段信息,其中,包括字段名、字段类型等信息
语法格式:

DESCRIBE 视图名; 
或者
DESC 视图名;
root@sso_db_69 17:58:  [escdb]> desc oa_org;
+-------------+--------------+------+-----+---------+-------+
| Field       | Type         | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+-------+
| id          | varchar(36)  | NO   |     | NULL    |       |
| name        | varchar(255) | NO   |     | NULL    |       |
| fullName    | varchar(255) | YES  |     | NULL    |       |
| resource_id | varchar(255) | YES  |     | NULL    |       |
+-------------+--------------+------+-----+---------+-------+
4 rows in set (0.01 sec)





---阅读剩余部分---

MySQL:Incorrect integer value: '' for column 'id' at row 1警告信息

MySQL5.x在insert into插入时,如果不带id写成为空如下:

INSERT INTO `server` VALUE ('',
'SSO系统',
'sso-db',
'192.168.6.37',
'/',
'运行',
'生产',
'4',
'8',
'100',
'CentOS Linux release 7.5',
'IDC机房',
'zuul',
'8000',
'/',
'王二小',
'2018-06-30',
server_update = NOW(),
'test11111111111111111111'
);

就会警告如下:

Warning Code : 1366
Incorrect integer value: '' for column 'id' at row 1

MySQL5的都会遇到这样的问题,插入空字符要使用NULL,改成如下:

INSERT INTO `server` VALUE (NULL,
'SSO系统',
'sso-db',
'192.168.6.37',
'/',
'运行',
'生产',
'4',
'8',
'100',
'CentOS Linux release 7.5',
'IDC机房',
'zuul',
'8000',
'/',
'王二小',
'2018-06-30',
server_update = NOW(),
'test11111111111111111111'
);

即可插入成功,同时也无警告信息。

MySQL5.7.5X在线调整innodb_buffer_pool_size参数

MySQL5.7.5以前,调整innodb_buffer_pool_size需要重启mysql进程才可以生效,建议业务低峰时间执行,在MySQL 5.7.5版本后,innodb_buffer_pool_size参数的值可以动态的设置,可以通过innodb_buffer_pool_chunk_size参数配置块的大小,Innodb_buffer_pool_resize_status状态变量记录了从调整操作的状态。

innodb_buffer_pool_size参数表示缓冲池字节大小,InnoDB缓存表和索引数据的内存区域。mysql默认的值是128M。最大值与你的CPU体系结构有关,在32位操作系统,最大值是 4294967295 (2^32-1) ,在64 位操作系统,最大值为18446744073709551615 (2^64-1)。在32位操作系统中,CPU和操作系统实用的最大大小低于设置的最大值。

innodb_log_file_size表示在一个日志组每个日志文件的字节大小。日志文件的总大小(innodb_log_file_size* innodb_log_files_in_group)不能超过最高值512GB。例如一对255 GB的日志文件,已经接近了极限,不能超过它。默认值是48M。比较合适的值的范围是从1MB到1 / N个的缓冲池大小,其中N是该组中的日志文件的数量。该值越大,缓冲池中必要的检查点刷新活动就会越少,节省磁盘I/ O。但是越大的日志文件,mysql的崩溃恢复就越慢,尽管在mysql5.5之后改进了恢复性能和日志文件恢复的代价,一般2G左右就可以了;

在线调整innodb_buffer_pool_size方法如下:
当前大小为128M

show variables like '%innodb_buffer_pool_size%';
+-------------------------+-----------+
| Variable_name           | Value     |
+-------------------------+-----------+
| innodb_buffer_pool_size | 134217728 |
+-------------------------+-----------+
row in set (0.03 sec)

select 134217728/1024/1024;  
+---------------------+
| 134217728/1024/1024 |
+---------------------+
|        128.00000000 |
+---------------------+
row in set (0.00 sec)

动态调整为512M

set global innodb_buffer_pool_size = 512*1024*1024;
Query OK, 0 rows affected (0.18 sec)

show variables like '%innodb_buffer_pool_size%';
+-------------------------+-----------+
| Variable_name           | Value     |
+-------------------------+-----------+
| innodb_buffer_pool_size | 536870912 |
+-------------------------+-----------+
row in set (0.02 sec)

select 536870912/1024/1024;
+---------------------+
| 536870912/1024/1024 |
+---------------------+
|        512.00000000 |
+---------------------+
1 row in set (0.00 sec)

调整时,内部会把数据页移动到一个新的位置,单位是块。如果想提升移动速度,则需要调整
innodb_buffer_pool_chunk_size的参数大小,默认是128M
innodb_buffer_pool_size/innodb_buffer_pool_instances = innodb_buffer_pool_chunk_size的参数大小,默认是128M
调整完以后在my.cnf的mysqld部分增加

innodb_buffer_pool_size = 536870912

以防止重启后仍然生效。

MySQL报错“Illegal mix of collations for operation 'like'”错误代码: 1271

在使用LIKE对日期或者时间字段进行模糊查询的时候,MySQL5.7X会报错误:

错误代码: 1271
Illegal mix of collations for operation 'like'

解决方法:在 MySQL5.5以上版本, 必需改成like binary '%中文%' ;
若字段 Type 是 time,date,datetime 在 select 時若使用 like '%中文%' 会出现 Illegal mix of collations for operation 'like'在写程序时要对每个字段进行搜索,在执行时可能就会出现时间字段 like '%中文%' 这种语法,
低版本MySQL是不会出现错误的。

使用sysbench对MySQL进行测试

sysbench是一个模块化的、跨平台、多线程基准测试工具,主要用于评估测试各种不同系统参数下的系统、数据库负载情况,项目地址:https://github.com/akopytov/sysbench
sysbench主要支持以下几种测试模式:
CPU运算性能
磁盘IO性能
调度程序性能
内存分配及传输速度
POSIX线程性能
数据库性能(OLTP基准测试)
目前sysbench主要支持 Mysql,Drizzle,PgSQL,Oracle等几种数据库。
快速安装:

yum -y install make automake libtool pkgconfig libaio-devel        
yum -y install mariadb-devel openssl-devel    # For MySQL support, replace with mysql-devel on RHEL/CentOS 5
yum -y install postgresql-devel        #For PostgreSQL support
curl -s https://packagecloud.io/install/repositories/akopytov/sysbench/script.rpm.sh | sudo bash
yum -y install sysbench

查看版本

[root@hongsin-monitor]# sysbench --version
sysbench 1.0.15

测试脚本路径

[root@hongsin-monitor sysbench]# pwd
/usr/share/sysbench
[root@hongsin-monitor sysbench]# ll
total 64
-rwxr-xr-x 1 root root  1452 Jul  4 04:06 bulk_insert.lua
-rw-r--r-- 1 root root 14369 Jul  4 04:06 oltp_common.lua
-rwxr-xr-x 1 root root  1290 Jul  4 04:06 oltp_delete.lua
-rwxr-xr-x 1 root root  2415 Jul  4 04:06 oltp_insert.lua
-rwxr-xr-x 1 root root  1265 Jul  4 04:06 oltp_point_select.lua
-rwxr-xr-x 1 root root  1649 Jul  4 04:06 oltp_read_only.lua
-rwxr-xr-x 1 root root  1824 Jul  4 04:06 oltp_read_write.lua
-rwxr-xr-x 1 root root  1118 Jul  4 04:06 oltp_update_index.lua
-rwxr-xr-x 1 root root  1127 Jul  4 04:06 oltp_update_non_index.lua
-rwxr-xr-x 1 root root  1440 Jul  4 04:06 oltp_write_only.lua
-rwxr-xr-x 1 root root  1919 Jul  4 04:06 select_random_points.lua
-rwxr-xr-x 1 root root  2118 Jul  4 04:06 select_random_ranges.lua
drwxr-xr-x 4 root root  4096 Oct 20 16:48 tests

写性能测试,数据准备

sysbench --test=/usr/share/sysbench/oltp_write_only.lua --mysql-host=192.168.255.90 --mysql-port=3306 --mysql-user=ice --mysql-password=xxxxxx --mysql-db=a1 --report-interval=10 --max-requests=0 --time=120 --threads=4500 --tables=10 --table-size=100000 prepare
WARNING: the --test option is deprecated. You can pass a script name or path on the command line without any options.
sysbench 1.0.15 (using bundled LuaJIT 2.1.0-beta2)

Initializing worker threads...

Creating table 'sbtest9'...Creating table 'sbtest8'...
Creating table 'sbtest3'...
Creating table 'sbtest4'...Creating table 'sbtest6'...


Creating table 'sbtest1'...
Creating table 'sbtest7'...
Creating table 'sbtest5'...
Creating table 'sbtest2'...
Creating table 'sbtest10'...
Inserting 100000 records into 'sbtest4'
Inserting 100000 records into 'sbtest10'
Inserting 100000 records into 'sbtest1'
Inserting 100000 records into 'sbtest2'
Inserting 100000 records into 'sbtest5'
Inserting 100000 records into 'sbtest8'
Inserting 100000 records into 'sbtest6'
Inserting 100000 records into 'sbtest9'
Inserting 100000 records into 'sbtest7'
Inserting 100000 records into 'sbtest3'
Creating a secondary index on 'sbtest10'...
Creating a secondary index on 'sbtest5'...
Creating a secondary index on 'sbtest8'...
Creating a secondary index on 'sbtest9'...
Creating a secondary index on 'sbtest6'...
Creating a secondary index on 'sbtest7'...
Creating a secondary index on 'sbtest1'...
Creating a secondary index on 'sbtest2'...
Creating a secondary index on 'sbtest4'...
Creating a secondary index on 'sbtest3'...
参数的解释:
--threads=4500  表示发起4500个并发连接
--oltp-read-only=off 表示不要进行只读测试,也就是会采用读写混合模式测试
--report-interval=10 表示每10秒输出一次测试进度报告
--rand-type=uniform 表示随机类型为固定模式,其他几个可选随机模式:uniform(固定),gaussian(高斯),special(特定的),pareto(帕累托)
--time=120 表示最大执行时长为 120秒
--max-requests=0 表示总请求数为 0,因为上面已经定义了总执行时长,所以总请求数可以设定为 0;也可以只设定总请求数,不设定最大执行时长
--tables=10  表示10个表
--table-size=100000  单表100000条记录
--percentile=99 表示设定采样比例,默认是 95%,即丢弃1%的长请求,在剩余的99%里取最大值

运行测试

sysbench --test=/usr/share/sysbench/oltp_write_only.lua --mysql-host=192.168.255.90 --mysql-port=3306 --mysql-user=ice --mysql-password=hongsinCS --mysql-db=a1 --report-interval=10 --max-requests=0 --time=120 --threads=4500 --tables=10 --table-size=100000 run
WARNING: the --test option is deprecated. You can pass a script name or path on the command line without any options.
sysbench 1.0.15 (using bundled LuaJIT 2.1.0-beta2)

Running the test with following options:
Number of threads: 4500
Report intermediate results every 10 second(s)
Initializing random number generator from current time


Initializing worker threads...

Threads started!

[ 10s ] thds: 4500 tps: 618.00 qps: 4951.50 (r/w/o: 0.00/1950.97/3000.53) lat (ms,95%): 8038.61 err/s: 91.70 reconn/s: 0.00
[ 20s ] thds: 4500 tps: 562.83 qps: 3806.01 (r/w/o: 0.00/1849.30/1956.71) lat (ms,95%): 15934.78 err/s: 227.01 reconn/s: 0.00
[ 30s ] thds: 4500 tps: 434.80 qps: 3051.53 (r/w/o: 0.00/1599.01/1452.51) lat (ms,95%): 22842.77 err/s: 226.20 reconn/s: 0.00
[ 40s ] thds: 4500 tps: 479.70 qps: 3310.41 (r/w/o: 0.00/1803.90/1506.50) lat (ms,95%): 28352.44 err/s: 224.60 reconn/s: 0.00
[ 50s ] thds: 4500 tps: 495.10 qps: 3403.11 (r/w/o: 0.00/1910.30/1492.80) lat (ms,95%): 28867.59 err/s: 219.60 reconn/s: 0.00
[ 60s ] thds: 4500 tps: 453.20 qps: 3140.89 (r/w/o: 0.00/1769.59/1371.29) lat (ms,95%): 28867.59 err/s: 215.80 reconn/s: 0.00
[ 70s ] thds: 4500 tps: 469.90 qps: 3246.01 (r/w/o: 0.00/1848.80/1397.20) lat (ms,95%): 28867.59 err/s: 228.90 reconn/s: 0.00
[ 80s ] thds: 4500 tps: 466.59 qps: 3238.67 (r/w/o: 0.00/1859.68/1378.99) lat (ms,95%): 29926.15 err/s: 225.20 reconn/s: 0.00
[ 90s ] thds: 4500 tps: 441.30 qps: 3084.43 (r/w/o: 0.00/1765.82/1318.61) lat (ms,95%): 29926.15 err/s: 225.30 reconn/s: 0.00
[ 100s ] thds: 4500 tps: 434.20 qps: 3026.20 (r/w/o: 0.00/1737.20/1289.00) lat (ms,95%): 29926.15 err/s: 217.40 reconn/s: 0.00
[ 110s ] thds: 4500 tps: 471.70 qps: 3247.39 (r/w/o: 0.00/1872.59/1374.80) lat (ms,95%): 32161.14 err/s: 218.40 reconn/s: 0.00
[ 120s ] thds: 4500 tps: 476.39 qps: 3289.44 (r/w/o: 0.00/1892.36/1397.07) lat (ms,95%): 31023.52 err/s: 232.80 reconn/s: 0.00
SQL statistics:
    queries performed:
        read:                            0         读操作
        write:                           233015     --写总数
        other:                           197026  --其他操作总数(SELECT、INSERT、UPDATE、DELETE之外的操作,例如COMMIT等)
        total:                           430041  --全部总数
    transactions:                        62617  (486.84 per sec.)  --总事务数(每秒事务数)
    queries:                             430041 (3343.55 per sec.) --查询数(每秒查询数)
    ignored errors:                      27087  (210.60 per sec.)  --忽略错误数
    reconnects:                          0      (0.00 per sec.)  --重新连接次数

General statistics:
    total time:                          128.6147s  --运行总时间
    total number of events:              62617  --事件总数

Latency (ms):
         min:                                    2.09
         avg:                                 8957.64
         max:                                99069.80
         95th percentile:                    27846.48
         sum:                            560900840.54

Threads fairness:    #线程平均数
    events (avg/stddev):           13.9149/3.92
    execution time (avg/stddev):   124.6446/2.39

清理数据:

sysbench --test=/usr/share/sysbench/oltp_write_only.lua --mysql-host=192.168.255.90 --mysql-port=3306 --mysql-user=ice --mysql-password=hongsinCS --mysql-db=a1 --report-interval=10 --max-requests=0 --time=120 --threads=4500 --tables=10 --table-size=100000 cleanup
WARNING: the --test option is deprecated. You can pass a script name or path on the command line without any options.
sysbench 1.0.15 (using bundled LuaJIT 2.1.0-beta2)

Dropping table 'sbtest1'...
Dropping table 'sbtest2'...
Dropping table 'sbtest3'...
Dropping table 'sbtest4'...
Dropping table 'sbtest5'...
Dropping table 'sbtest6'...
Dropping table 'sbtest7'...
Dropping table 'sbtest8'...
Dropping table 'sbtest9'...
Dropping table 'sbtest10'...

可以根据/usr/share/sysbench下面的lua脚本进行读、写、更新,随机等等测试,条件允许建议时间1小时以上,以便测试数据准确·

mysqlslap: [ERROR] unknown variable 'default-character-set=utf8mb4'

运行mysqlslap报错如下:

[root@monitor-db ~]# mysqlslap --version
mysqlslap: Error when connecting to server: Access denied for user 'root'@'localhost' (using password: NO)

解决方法:带上--no-defaults参数

[root@sso-db-a binlog]# mysqlslap --no-defaults --version
mysqlslap  Ver 1.0 Distrib 5.7.23, for Linux (x86_64)

或者在my.cnf里面将default-character-set修改为utf8,需重启生效

[client]
#设置MySQL客户端的字符集
default-character-set=utf8

mysqlslap.png

FATAL: MySQL error: 1461 "Can't create more than max_prepared_stmt_count statements (current value: 16382)"

使用sysbench做MySQL性能压测的时候,报错:

FATAL: mysql_stmt_prepare() failed
(last message repeated 2 times)
FATAL: MySQL error: 1461 "Can't create more than max_prepared_stmt_count statements (current value: 16382)"
(last message repeated 1 times)

max_prepared_stmt_count参数限制了同一时间在mysqld上所有session中prepared 语句的上限。
它的取值范围为“0 - 1048576”,默认为16382。
mysql对于超出max_prepared_stmt_count的prepare语句就会报1461的错误。
sysvar_max_prepared_stmt_count.png
官方文档:https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html
一般而言,默认值应该是足够用的,因为现场的并发其实没有那么的大。
一个可能的原因是应用端那边没有关闭prepared的语句。
直连后端master执行如下命令
mysql> show global status like ‘com_stmt%’;
查看如下3个参数值:
Com_stmt_close prepare语句关闭的次数
Com_stmt_execute prepare语句执行的次数
Com_stmt_prepare prepare语句创建的次数

请确认Com_stmt_close的值是否接近于Com_stmt_prepare。

mysql> show global status like 'com_stmt%';
+-------------------------+----------+
| Variable_name           | Value    |
+-------------------------+----------+
| Com_stmt_execute        | 79358467 |
| Com_stmt_close          | 18100    |
| Com_stmt_fetch          | 0        |
| Com_stmt_prepare        | 67646    |
| Com_stmt_reset          | 0        |
| Com_stmt_send_long_data | 0        |
| Com_stmt_reprepare      | 0        |
+-------------------------+----------+
7 rows in set (0.00 sec)
mysql> show global variables like 'max_prepared_stmt_count';
+-------------------------+-------+
| Variable_name           | Value |
+-------------------------+-------+
| max_prepared_stmt_count | 16382 |
+-------------------------+-------+
1 row in set (0.01 sec)
mysql> set global max_prepared_stmt_count=1048576;
Query OK, 0 rows affected (0.00 sec)
mysql> show global variables like 'max_prepared_stmt_count';
+-------------------------+-------+
| Variable_name           | Value |
+-------------------------+-------+
| max_prepared_stmt_count |1048576|
+-------------------------+-------+
1 row in set (0.00 sec)

修改之后再次使用100并发压测则没有报错了。
1 row in set (0.01 sec)
为保证mysql实例重启生效可以写到配置文件my.cnf

[mysqld]
max_prepared_stmt_count=1048576

log_error_verbosity日志参数

MySQL中log_error定义是否启用错误日志的功能和错误日志的存储位置,log_warnings定义是否将告警信息(warning messages)也写入错误日志。此选项默认启用,具体来说:

log_warnings 为0, 表示不记录告警信息。
log_warnings 为1, 表示告警信息写入错误日志。
log_warnings 大于1, 表示各类告警信息,例如有关网络故障的信息和重新连接信息写入错误日志。

此参数在不同版本略有差别,在MySQL 5.6中,log_warnings的默认值为1
在MySQL 5.7中,有些版本默认值为2,有些版本默认值为1, 具体参考官方文档信息,如下所示:
5.7.png
从MySQL 5.7.2开始,首选log_error_verbosity系统变量,而不是使用--log-warnings选项或log_warnings系统变量,这个参数从MySQL 8.0.3开始被移除了:
This system variable was removed in MySQL 8.0.3. Use the log_error_verbosity system variable instead.

新参数log_error_verbosity更简单,它有三个可选值, 分别对应:1 错误信息;2 错误信息和告警信息; 3:错误信息、告警信息和通知信息。 具体参考官方文档,下面部分截取官方文档。
8.0.png
查看错误log设置:

root@sso_db_69 15:34:  [(none)]> show variables like '%log_error%';
+---------------------+-----------------------+
| Variable_name       | Value                 |
+---------------------+-----------------------+
| binlog_error_action | ABORT_SERVER          |
| log_error           | /data/mysql/error.log |
| log_error_verbosity | 2                     |
+---------------------+-----------------------+
3 rows in set (0.00 sec)

调整log_error_verbosity级别:

SET GLOBAL log_warnings=2;
SET GLOBAL log_error_verbosity=2;

永久调整my.cnf增加设置:

[mysqld]
log_warnings=1
log_error_verbosity=1

参数说明

Binlog_error_action=ABORT_SERVER
Binlog_error_action参数控制当不能写binlog时,mysql-server将会采取什么行动。
设置binlog_error_action=ABORT_SERVER会使mysql-server在写binlog遇到严重错误时退出,比如磁盘满了,文件系统不可写入了等。
在ABORT_SERVER选项下,binlog和从库都是安全的,这是官方修改此默认值的原因。
在先前的选项下(binlog_error_action=IGNORE_ERROR),如果一个错误发生,导致无法写入binlog,mysql-server会在错误日志中记录错误并强制关闭binlog功能。这会使mysql-server在不记录binlog的模式下继续运行,导致从库无法继续获取到主库的binlog。

官方参考资料:
https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_log_warnings
https://dev.mysql.com/doc/refman/5.7/en/server-options.html#option_mysqld_log-warnings

MySQL全局锁库锁表

1、全局读锁定:

FLUSH TABLES WITH READ LOCK ;

执行了命令之后所有库所有表都被锁定只读,一般用在数据库联机备份,这个时候数据库的写操作将被阻塞,读操作顺利进行。

解锁:

UNLOCK TABLES ;

2、全局表级别锁定:

LOCK TABLES tbl_name ; #不影响其他表的写操作

解锁也是:

UNLOCK TABLES ;

这两个语句在执行的时候都需要注意个特点,就是隐式提交的语句,在退出mysql终端的时候都会隐式的执行unlock tables,也就是如果要让表锁定生效就必须一直保持对话。

read-lock:  允许其他并发的读请求,但阻塞写请求,即可以同时读,但不允许任何写。也叫共享锁
write-lock: 不允许其他并发的读和写请求,是排他的(exclusive)。也叫独占锁

MySQL删除数据库中的所有表的两个方法

快速删除MySQL数据库中的所有表的两个方法:

1、最简单的方法,最方便的方法:

删除数据库,然后重新建立一个空数据库

2、但是有删除所有表的方法

(1)使用concat函数产生删除表的sql语句,

select concat("DROP TABLE IF EXISTS ", table_name, ";") from information_schema.tables where table_schema="Your_database_name";

(2)执行sql语句

DROP TABLE IF EXISTS aws_active_data;
DROP TABLE IF EXISTS aws_answer;
DROP TABLE IF EXISTS aws_answer_comments;
DROP TABLE IF EXISTS aws_answer_thanks;
DROP TABLE IF EXISTS aws_answer_uninterested;
DROP TABLE IF EXISTS aws_answer_vote;
DROP TABLE IF EXISTS aws_approval;
DROP TABLE IF EXISTS aws_article;
DROP TABLE IF EXISTS aws_article_comments;
DROP TABLE IF EXISTS aws_article_vote;
DROP TABLE IF EXISTS aws_attach;
DROP TABLE IF EXISTS aws_category;
DROP TABLE IF EXISTS aws_column;
DROP TABLE IF EXISTS aws_column_focus;
DROP TABLE IF EXISTS aws_draft;
DROP TABLE IF EXISTS aws_edm_task;
DROP TABLE IF EXISTS aws_edm_taskdata;
DROP TABLE IF EXISTS aws_edm_unsubscription;
DROP TABLE IF EXISTS aws_edm_userdata;
DROP TABLE IF EXISTS aws_edm_usergroup;
DROP TABLE IF EXISTS aws_education_experience;
DROP TABLE IF EXISTS aws_favorite;

附concat函数使用方法:
CONCAT(str1,str2,…)

返回结果为连接参数产生的字符串。如有任何一个参数为NULL ,则返回值为 NULL。

注意:
如果所有参数均为非二进制字符串,则结果为非二进制字符串。
如果自变量中含有任一二进制字符串,则结果为一个二进制字符串。
一个数字参数被转化为与之相等的二进制字符串格式;若要避免这种情况,可使用显式类型 cast, 例如:

SELECT CONCAT(CAST(int_col AS CHAR), char_col)

MySQL的concat函数可以连接一个或者多个字符串,如

mysql> select concat('10');
+--------------+
| concat('10') |
+--------------+
| 10   |
+--------------+
1 row in set (0.00 sec)
mysql> select concat('11','22','33');
+------------------------+
| concat('11','22','33') |
+------------------------+
| 112233 |
+------------------------+
1 row in set (0.00 sec)

MySQL的concat函数在连接字符串的时候,只要其中一个是NULL,那么将返回NULL
mysql> select concat('11','22',null);
+------------------------+
| concat('11','22',null) |
+------------------------+
| NULL   |
+------------------------+
1 row in set (0.00 sec)

Mysql 查看连接数、状态、最大并发数

MySQL: ERROR 1040: Too many connections”的异常情况,造成这种情况的一种原因是访问量过高,MySQL服务器抗不住,这个时候就要考虑增加从服务器分散读压力;另一种原因就是MySQL配置文件中max_connections值过小。
首先,我们来查看mysql的最大连接数:

root@hongsin-monitor-db 18:14:58 [(none)]>show variables like '%max_connections%';
+-----------------+--------+
| Variable_name   | Value  |
+-----------------+--------+
| max_connections | 100000 |
+-----------------+--------+
1 row in set (0.00 sec)

查看服务器响应的最大连接数:

root@hongsin-monitor-db 18:11:29 [(none)]> show global status like 'Max_used_connections';
+----------------------+-------+
| Variable_name        | Value |
+----------------------+-------+
| Max_used_connections | 139   |
+----------------------+-------+
1 row in set (0.01 sec)

可以看到服务器响应的最大连接数为139,远远低于mysql服务器允许的最大连接数值。

对于mysql服务器最大连接数值的设置范围比较理想的是:服务器响应的最大连接数值占服务器上限连接数值的比例值在10%以上,如果在10%以下,说明mysql服务器最大连接上限值设置过高.

Max_used_connections / max_connections * 100% = 139/100000 *100% ≈ 0.139%

我们可以看到占比远低于10%(因为这是本地监控测试服务器,结果值没有太大的参考意义,大家可以根据实际情况设置连接数的上限值)。
设置这个最大连接数值
方法1:

set GLOBAL max_connections=256;

Query OK, 0 rows affected (0.00 sec)

mysql> show variables like '%max_connections%';

+-----------------+-------+

| Variable_name  | Value |

+-----------------+-------+

| max_connections | 256  |

+-----------------+-------+

1 row in set (0.00 sec)

方法2:

修改mysql配置文件my.cnf,在[mysqld]段中添加或修改max_connections值:
max_connections=256

重启mysql服务即可。

Mysql5.5 mysql5.6 mysql5.7:默认的最大连接数都是151,上限为:100000
Mysql5.0版本:默认的最大连接数为100,上限为16384

Zabbix3.4.11使用自带模板监控MySQL

Zabbix默认的agentd模板里面有一个userparameter_mysql.conf配置文件,修改文件内容如下:

[root@c75 zabbix_agentd]# cat /usr/local/zabbix/conf/zabbix_agentd/userparameter_mysql.conf 
UserParameter=mysql.status[*],echo "show global status where Variable_name='$1';" | HOME=/usr/local/zabbix/conf/ mysql -N | awk '{print $$2}'
UserParameter=mysql.size[*],bash -c 'echo "select sum($(case "$3" in both|"") echo "data_length+index_length";; data|index) echo "$3_length";; free) echo "data_free";; esac)) from information_schema.tables$([[ "$1" = "all" || ! "$1" ]] || echo " where table_schema=\"$1\"")$([[ "$2" = "all" || ! "$2" ]] || echo "and table_name=\"$2\"");" | HOME=/usr/local/zabbix/conf/ mysql -N'
UserParameter=mysql.ping,netstat -ntpl |grep 3306 |grep mysql |wc|awk '{print $1}'
UserParameter=mysql.version,mysql -V
UserParameter=mysql.vars[*],echo "show variables where Variable_name='$1';" | HOME=/usr/local/zabbix/conf/ mysql -N | awk '{print $$2}'

做下MySQL的软连接:

ln -s /usr/local/mysql/bin/mysql /bin/

修改zabbix_agentd.conf文件,增加

Include=/usr/local/zabbix/conf/zabbix_agentd/*.conf

登陆MySQL控制台创建只读用户:

GRANT SELECT ON *.* TO 'zabbix'@'localhost' IDENTIFIED BY 'xxxxxx';

在/usr/local/zabbix/conf创建.my.cnf文件,内容如下:

[root@c75 conf]# cat /usr/local/zabbix/conf/.my.cnf 
# Zabbix Agent
[mysql]
host=localhost
user=zabbix
password="xxxxxx"
socket=/data/mysql/mysql.sock
[mysqladmin]
host=localhost
user=zabbix
password="xxxxxx"
socket=/data/mysql/mysql.sock

重启zabbix-agentd

killall -9 zabbix_agentd
/usr/local/zabbix/sbin/zabbix_agentd -c /usr/local/zabbix/conf/zabbix_agentd.conf

在zabbix-server机器上检测

[root@monitor ~]# zabbix_get -s 192.168.3.244 -p 10050 -k mysql.version
mysql  Ver 14.14 Distrib 5.7.22, for Linux (x86_64) using  EditLine wrapper
[root@monitor ~]# zabbix_get -s 192.168.3.244 -p 10050 -k mysql.ping
1

注意:如果测试的时候报错:sh: mysql: command not found,sh: mysqladmin: command not found,
修改userparameter_mysql.conf中mysql,mysqladmin命令的路径,使用绝对路径或者做下软连接即可;
如果报错

(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.

执行的脚本中执行

chmod +s /bin/netstat

来解决chmod +s 的意思是:为了方便普通用户执行一些特权命令,SUID/SGID程序允许普通用户以root身份暂时执行该程序,并在执行结束后再恢复身份
给要监控的主机添加MySQL模板,最后效果如下图:
1.png
2.png

MySQL报错This function has none of DETERMINISTIC解决

创建存储过程出错log

ERROR 1418 (HY000): This function has none of DETERMINISTIC, NO SQL, or READS SQL DATA in its declaration and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable)

原因:开启了bin-log就须指函数是否是

1 DETERMINISTIC 不确定的

2 NO SQL 没有SQl语句,当然也不会修改数据

3 READS SQL DATA 只是读取数据,当然也不会修改数据

4 MODIFIES SQL DATA 要修改数据

5 CONTAINS SQL 包含了SQL语句

其中在function里面,只有 DETERMINISTIC, NO SQL 和 READS SQL DATA 被支持。
如果开启了 bin-log, 我们就必须为function指定一个参数。
临时设置:

mysql> show variables like 'log_bin_trust_function_creators';
+---------------------------------+-------+
| Variable_name          | Value |
+---------------------------------+-------+
| log_bin_trust_function_creators | OFF  |
+---------------------------------+-------+
mysql> set global log_bin_trust_function_creators=1;
mysql> show variables like 'log_bin_trust_function_creators';
+---------------------------------+-------+
| Variable_name          | Value |
+---------------------------------+-------+
| log_bin_trust_function_creators | ON  |

修改配置文件my.cnf在mysqld部分增加

log_bin_trust_function_creators=1
/etc/init.d/mysql restart

重启MySQL服务。

最新

分类

归档

评论

其它