Fork me on GitHub

发表于 | 分类于 |

错误：

1
2
3
java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
Job Submission failed with exception 'org.apache.hadoop.security.AccessControlException(Permission denied: user=anonymous, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x

解决：
权限问题

1
hadoop fs -chmod -R 777  /

阅读全文 »

发表于 | 分类于 |

Coding.net����

�˻� SSH ��Կ

�˻� SSH ��Կ�Ǹ��û��˻������Ĺ�Կ��һ�����ã�SSH ��ӵ���˻���������Ŀ�ֿ��Ķ�дȨ�ޡ� ���á��˻� SSH ��Կ���ǿ�����ʹ�� SSH ��ʽ����/�޸Ĵ����ֿ��ġ�ǰ�ù���������Ϊ����ȡ SSH Э����ַ���������ɹ�Կ�������� Coding.net ���ӹ�Կ���������衣

阅读全文 »

发表于 | 分类于 |

已安装Hadoop2.7.2的三节点集群

安装Scala

  1. 下载scala-2.11.7.tgz

  2. 解压Scala

    1
    $ tar -zxvf scala-2.11.7.tgz -C ~/cloud/

  3. 配置环境变量

    1
    2
    export  SCALA_HOME=/home/ubuntu/cloud/scala-2.11.7
    export PATH = $PATH:$SCALA_HOME/bin

  4. 测试

    1
    2
    scala -version
    Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL

  5. 配置到每台节点

    阅读全文 »

发表于 | 分类于 |

name node is in safe mode

问题： 向hdfs put数据的时候，导致了 name node is in safe mode，然后使用 Hadoop dfsadmin -safemode leave 后， 解除了安全模式。可是再次使用hdfs put或rm数据，仍旧导致name node 进入安全模式。

答案：分析了一下，问题是namenode所在机器的硬盘满了。因此即使使用了 hadoop dfsadmin -safemode leave 之后， 仍旧不能使用hdfs。

解决办法：

  1. 删除namenode所在机器的一些数据（本地数据）
  2. 结束安全模式 hadoop dfsadmin -safemode leave
  3. 可以正常使用hdfs了

发表于 | 分类于 |

LinkedHashMap和TreeMap的区别

首先2个都是map，所以用key取值肯定是没区别的，区别在于用Iterator遍历的时候
LinkedHashMap保存了记录的插入顺序，先插入的先遍历到
TreeMap默认是按升序排，也可以指定排序的比较器。遍历的时候按升序遍历。
例如：a是LinkedHashMap，b是TreeMap。
a.put(“2”,”ab”);
a.put(“1”,”bc”);
b.put(“2”,”ab”);
b.put(“1”,”bc”);

那么遍历a的时候，先遍历到key是2的，因为2先放进去。
遍历b的时候，先遍历到“1”，因为按顺序是先1后2

发表于 | 分类于 |

题目：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
To and Fro
Time Limit: 2000/1000 MS (Java/Others)    Memory Limit: 65536/32768 K (Java/Others)
Total Submission(s): 6390    Accepted Submission(s): 4389
Problem Description
Mo and Larry have devised a way of encrypting messages. They first decide secretly on the number of columns and write the message (letters only) down the columns, padding with extra random letters so as to make a rectangular array of letters. For example, if the message is “There’s no place like home on a snowy night” and there are five columns, Mo would write down
t o i o y
h p k n n
e l e a i
r a h s g
e c o n h
s e m o t
n l e w x
Note that Mo includes only letters and writes them all in lower case. In this example, Mo used the character ‘x’ to pad the message out to make a rectangle, although he could have used any letter.
Mo then sends the message to Larry by writing the letters in each row, alternating left-to-right and right-to-left. So, the above would be encrypted as
toioynnkpheleaigshareconhtomesnlewx
Your job is to recover for Larry the original message (along with any extra padding letters) from the encrypted one.
 
Input
There will be multiple input sets. Input for each set will consist of two lines. The first line will contain an integer in the range 2. . . 20 indicating the number of columns used. The next line is a string of up to 200 lower case letters. The last input set is followed by a line containing a single 0, indicating end of input.
 
Output
Each input set should generate one line of output, giving the original plaintext message, with no spaces.
 
Sample Input
5
toioynnkpheleaigshareconhtomesnlewx
3
ttyohhieneesiaabss
0
 
Sample Output
theresnoplacelikehomeonasnowynightx
thisistheeasyoneab
 
Source
East Central North America 2004
 
Recommend
Ignatius.L   |   We have carefully selected several similar problems for you:  1196 1073 1161 1113 1256

解答：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
package hdu;
import java.util.Scanner;
/**
 * 做的时候没看清楚  直接把蛇形输出做了正序的转换，然后才进行二维数组存储，多了个步骤
 * @author Skye
 *
 */
public class Main {
	public static void main(String[] args) {
		// TODO Auto-generated method stub
		Scanner sc = new Scanner(System.in);
		int col = sc.nextInt();
		String message = null;
		int rowNum,row = 0;
		StringBuffer result = new StringBuffer();
		char[] stringArr = null;
		StringBuffer tmpString = new StringBuffer();
		char[] tmpChar = null;
		boolean isChange = false;
		while (col != 0) {
			message = sc.next();
			rowNum = message.length()/col;
			// System.out.println(message);
			stringArr = message.toCharArray();
			// ttyohhieneesiaabss
			for (int i = 0; i < stringArr.length; i++) {
				// System.out.println(stringArr[i]);
				if (i % col == 0) { // 到行开头
					row = i / col + 1; // 记录当前字母在第几行
					if (row % 2 == 0) {
						isChange = true;
					} else {
						isChange = false;
					}
				}
				if (isChange == false) {
					
					if (i != 0 && i % col == 0) {
						//System.out.println(tmpString.toString());
						tmpChar = tmpString.toString().toCharArray();
						for (int j = tmpChar.length - 1; j >= 0; j--) {
							result.append(tmpChar[j]);
						}
						tmpString = new StringBuffer();
					}
					result.append(stringArr[i]);
				} else {
					tmpString.append(stringArr[i]);
				}
			}
			if(isChange == true){
				//System.out.println(tmpString.toString());
				tmpChar = tmpString.toString().toCharArray();
				for (int j = tmpChar.length - 1; j >= 0; j--) {
					result.append(tmpChar[j]);
				}
				tmpString = new StringBuffer();
			}
			//System.out.println(result);
			
			char[][] charArr = new char[rowNum][col];
			char[] resultChar = result.toString().toCharArray();
			int count = 0;
			for(int k = 0; k < rowNum;k++){
				for(int p = 0; p < col;p++){
					charArr[k][p] = resultChar[count++];
				}
			}
			StringBuffer resultFinal = new StringBuffer();
			for(int k = 0; k < col;k++){
				for(int p = 0; p < rowNum;p++){
					resultFinal.append(charArr[p][k]);
				}
			}
			System.out.println(resultFinal);
			resultFinal = new StringBuffer();
			result = new StringBuffer();
			col = sc.nextInt();
		}
		sc.close();
	}
}

HDU另解：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
import java.io.*;
import java.util.*;
public class Main
{
	public static void main(String[] args)
	{
		Scanner input = new Scanner(System.in);
		while (input.hasNext())
		{
			int n = input.nextInt();
			if(n==0) break;
			input.nextLine();
			String str = input.nextLine();
			char c1[] = str.toCharArray();                   // 把每个字符单个存起来
			char c2[][] = new char[1010][1010];
			int line = c1.length / n;                        //记录行数
			int k = 0;                                      // 记录c1字符的位数
			for (int i = 0; i < line; i++)
			{
				if (i % 2 == 1)
				{
					for (int j = n - 1; j >= 0; j--)
					{
						c2[i][j] = c1[k++];
					}
				} 
				else
				{
					for (int j = 0; j < n; j++)
					{
						c2[i][j] = c1[k++];
					}
				}
			}
			for (int j = 0; j < n; j++)
			{
				for (int i = 0; i < line; i++)
				{
					System.out.print(c2[i][j]);
				}
			}
			System.out.println();
		}
	}
}

发表于 | 分类于 |

做推荐系统项目时，对标签评分需要对标签评分map进行排序.

理论准备

  • Map是键值对的集合接口，它的实现类主要包括：HashMap,TreeMap,Hashtable以及LinkedHashMap等。

  • TreeMap：基于红黑树（Red-Black tree）的 NavigableMap 实现，该映射根据其键的自然顺序进行排序，或者根据创建映射时提供的 Comparator 进行排序，具体取决于使用的构造方法。
    HashMap的值是没有顺序的，它是按照key的HashCode来实现的，对于这个无序的HashMap我们要怎么来实现排序呢？参照TreeMap的value排序。

  • Map.Entry返回Collections视图。

key排序

TreeMap默认是升序的，如果我们需要改变排序方式，则需要使用比较器：Comparator。Comparator可以对集合对象或者数组进行排序的比较器接口，实现该接口的public compare(T o1,To2)方法即可实现排序，如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import java.util.Comparator;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;
public class TreeMapTest {
    public static void main(String[] args) {
        Map<String, String> map = new TreeMap<String, String>(
                new Comparator<String>() {
                    public int compare(String obj1, String obj2) {
                        // 降序排序
                        return obj2.compareTo(obj1);
                    }
                });
        map.put("b", "ccccc");
        map.put("d", "aaaaa");
        map.put("c", "bbbbb");
        map.put("a", "ddddd");
        
        Set<String> keySet = map.keySet();
        Iterator<String> iter = keySet.iterator();
        while (iter.hasNext()) {
            String key = iter.next();
            System.out.println(key + ":" + map.get(key));
        }
    }
}

运行如下：

1
2
3
4
d:aaaaa
c:bbbbb
b:ccccc
a:ddddd

value排序

上面例子是对根据TreeMap的key值来进行排序的，但是有时我们需要根据TreeMap的value来进行排序。对value排序我们就需要借助于Collections的sort(List list, Comparator<? super T> c)方法，该方法根据指定比较器产生的顺序对指定列表进行排序。但是有一个前提条件，那就是所有的元素都必须能够根据所提供的比较器来进行比较，如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.TreeMap;
public class TreeMapTest {
    public static void main(String[] args) {
        Map<String, String> map = new TreeMap<String, String>();
        map.put("a", "ddddd");
        map.put("c", "bbbbb");
        map.put("d", "aaaaa");
        map.put("b", "ccccc");
        
        //这里将map.entrySet()转换成list
        List<Map.Entry<String,String>> list = new ArrayList<Map.Entry<String,String>>(map.entrySet());
        //然后通过比较器来实现排序
        Collections.sort(list,new Comparator<Map.Entry<String,String>>() {
            //升序排序
            public int compare(Entry<String, String> o1,
                    Entry<String, String> o2) {
                return o1.getValue().compareTo(o2.getValue());
            }
            
        });
        
        for(Map.Entry<String,String> mapping:list){ 
               System.out.println(mapping.getKey()+":"+mapping.getValue()); 
          } 
    }
}

运行结果如下：

1
2
3
4
d:aaaaa
c:bbbbb
b:ccccc
a:ddddd

参考：http://blog.csdn.net/xiaoyu714543065/article/details/38519817

发表于 | 分类于 |

javaBufferdReader读取文件乱码

以下为读取文件方法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
private static void putIdGame(){
	
	URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
	int count = 0;
	try {
		
		URL url = new URL(HDFS + path);
		InputStream gameList = url.openStream();
		BufferedReader reader_url = new BufferedReader(new InputStreamReader(gameList,"UTF-8"));
		String inString_RL = reader_url.readLine();
		
		while (inString_RL != null && count < 50) {
			int userId;
			String[] str = inString_RL.split(","); 
			count ++;
			map.put(str[1], str[0]);
			System.out.println(str[0]);
			inString_RL = reader_url.readLine();
		}
		reader_url.close();
	} catch (FileNotFoundException e) {
		System.out.println("未找文件！");
	} catch (IOException e1) {
		System.out.println("文件读写错误！");
	}
}

在InputStreamReader中加入”UTF-8”即可

Java读取文件时第一行出现乱码“？”问号

在windows 环境下，使用java文件流读取文本文件时，会出现第一个字符为未知字符”?” ,其他字符完整。而且第一个字符显示为？但是用equals比对发现并非是”?”号,google之，了解到bom编码标记。使用 16进制打印输出结果：

只要出现该头的16进制编码为这种字符便可以断定该文本文件的编码方式了。

bom编码标记：

bom全称是：byte order mark，汉语意思是标记字节顺序码。只是出现在：unicode字符集中，只有unicode字符集，存储时候，要求指定编码，如果不指定，windows还会用默认的：ANSI读取。常见的bom头是：

UTF-8 ║ EF BB BF
UTF-16LE ║ FF FE (小尾）
UTF-16BE ║ FE FF （大尾）
UTF-32LE ║ FF FE 00 00
UTF-32BE ║ 00 00 FE FF

解决方法：

  1. 工具将txt文件另存为UTF-8无BOM格式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
public String readerFile(InputStream in) throws IOException {
		StringBuffer strBuff = new StringBuffer();
		String temp = null;
		BufferedReader reader = new BufferedReader(new InputStreamReader(in,Charset.forName("utf-8")));
		while ((temp = reader.readLine()) != null) {
			byte[] by = temp.getBytes();
			String header = Integer.toHexString(by[0]).toUpperCase();
			//判断是否拥有无法识别的字符
			if (header.equalsIgnoreCase("FFFFFFEF") || header.equalsIgnoreCase("3F")) {
				strBuff.append(temp.substring(1) + "\n");
				continue;
			}
			strBuff.append(temp + "\n");
		}
		reader.close();
		in.close();
		return strBuff.toString();
	}

发表于 | 分类于 |

花了点时间整理了验证一下在本地eclipse上操作HDFS的工具类，实现在本地通过API操作HDFS。

实现以下功能：

  • ls
  • rmr
  • mkdir
  • copyFromLocal
  • cat
  • copyToLocal
  • 创建一个新文件，并写入内容

主要引用参考：

http://blog.fens.me/hadoop-hdfs-api/

类如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
package com.hdu.hdfs;
import java.io.IOException;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.BlockLocation;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Hdfs;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.web.HsftpFileSystem;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.mapred.JobConf;
/**
 * HDFS工具类
 * 
 * 实现功能：
 * hadoop fs -ls / 
 * hadoop fs -mkdir /data 
 * hadoop fs -rmr /data/test.txt
 * hadoop fs -copyFromLocal /test/test.txt /data 
 * hadoop fs -cat /data/test.txt
 * hadoop fs -copyToLocal /data /test/test.txt 
 * 创建一个新文件，并写入内容  
 * 重命名
 * 
 * 需要导入以下路径的所有jar包： hadoop-2.7.2\share\hadoop\common
 * hadoop-2.7.2\share\hadoop\common\lib hadoop-2.7.2\share\hadoop\hdfs
 * hadoop-2.7.2\share\hadoop\hdfs\lib hadoop-2.7.2\share\hadoop\mapreduce
 * 
 * 
 * @author Skye
 *
 */
public class HdfsDAO {
	// HDFS访问地址
	private static final String HDFS = "hdfs://192.168.1.111:9000/";
	// hdfs路径
	private String hdfsPath;
	// Hadoop系统配置
	private Configuration conf;
	public HdfsDAO(Configuration conf) {
		this(HDFS, conf);
	}
	public HdfsDAO(String hdfs, Configuration conf) {
		this.hdfsPath = hdfs;
		this.conf = conf;
	}
	// 启动函数
	public static void main(String[] args) throws IOException {
		JobConf conf = config();
		HdfsDAO hdfs = new HdfsDAO(conf);
		// hdfs.mkdirs("/new");
		// 可以同时建多级目录
		// hdfs.mkdirs("/new/new3");
		// hdfs.ls("/tuijian");
		// hdfs.rmr("/new");
		// 可用当前eclipse工作空间的相对路径和文件绝对路径 以及当前项目的路径不加"/"
		// hdfs.copyFileToHdfs("data/hive笔记.md", "/data");
		// hdfs.copyFileToHdfs("/Xiaomi/MiFlashClean.cmd", "/data");
		// hdfs.copyFileToHdfs("E:/推荐系统/100万用户数据/user_pay", "/data");
		// hdfs.rmr("/data/MiFlashClean.cmd");
		// hdfs.rmr("/data/user_pay_201606");
		// hdfs.createFile("/new/createTest", "1,英雄联盟");
		// hdfs.download("/data/RecommendList", "C:/Users/Skye/Desktop");
		// hdfs.cat("/data/RecommendList1");
		// hdfs.renameFile("/data/RecommendList", "/data/RecommendListOld");
		// hdfs.ls("/data");
		//hdfs.findLocationOnHadoop("/data/RecommendListOld");
	}
	// 加载Hadoop配置文件
	public static JobConf config() {
		JobConf conf = new JobConf(HdfsDAO.class);
		conf.setJobName("HdfsDAO");
		// conf.addResource("classpath:/hadoop/core-site.xml");
		// conf.addResource("classpath:/hadoop/hdfs-site.xml");
		// conf.addResource("classpath:/hadoop/mapred-site.xml");
		return conf;
	}
	public void mkdirs(String folder) throws IOException {
		Path path = new Path(folder);
		FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf);
		if (!fs.exists(path)) {
			fs.mkdirs(path);
			System.out.println("Create: " + folder);
		}
		fs.close();
	}
	public void rmr(String folder) throws IOException {
		Path path = new Path(folder);
		FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf);
		fs.deleteOnExit(path);
		System.out.println("Delete: " + folder);
		fs.close();
	}
	public void ls(String folder) throws IOException {
		Path path = new Path(folder);
		FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf);
		FileStatus[] list = fs.listStatus(path);
		System.out.println("ls: " + folder);
		System.out.println("==========================================================");
		for (FileStatus f : list) {
			System.out.printf("name: %s, folder: %s, size: %d\n", f.getPath(), f.isDir(), f.getLen());
		}
		System.out.println("==========================================================");
		fs.close();
	}
	public void createFile(String file, String content) throws IOException {
		FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf);
		byte[] buff = content.getBytes();
		FSDataOutputStream os = null;
		try {
			os = fs.create(new Path(file));
			os.write(buff, 0, buff.length);
			System.out.println("Create: " + file);
		} finally {
			if (os != null)
				os.close();
		}
		fs.close();
	}
	public void copyFileToHdfs(String local, String remote) throws IOException {
		FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf);
		fs.copyFromLocalFile(new Path(local), new Path(remote));
		System.out.println("copy from: " + local + " to " + remote);
		fs.close();
	}
	public void download(String remote, String local) throws IOException {
		Path path = new Path(remote);
		FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf);
		fs.copyToLocalFile(path, new Path(local));
		System.out.println("download: from" + remote + " to " + local);
		fs.close();
	}
	public void renameFile(String oldFileName, String newFileName) throws IOException {
		boolean isSuccess = true;
		FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf);
		try {
			isSuccess = fs.rename(new Path(oldFileName), new Path(newFileName));
		} catch (IOException e) {
			isSuccess = false;
		}
		System.out.println(isSuccess ? "Rename success！ " + oldFileName + " to " + newFileName
				: "Rename failed！" + oldFileName + " to " + newFileName);
		fs.close();
	}
	/**
	 * 查看某个文件在HDFS集群的位置
	 * 
	 * @throws IOException
	 */
	public void findLocationOnHadoop(String filePath) throws IOException {
		// Path targetFile=new Path(rootPath+"user/hdfsupload/AA.txt");
		// FileStatus fileStaus=coreSys.getFileStatus(targetFile);
		Path targetFile = new Path(HDFS + filePath);
		FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf);
		FileStatus fileStaus = fs.getFileStatus(targetFile);
		BlockLocation[] bloLocations = fs.getFileBlockLocations(fileStaus, 0, fileStaus.getLen());
		for (int i = 0; i < bloLocations.length; i++) {
			System.out.println("block_" + i + "_location:" + bloLocations[i].getHosts()[0]);
		}
		fs.close();
	}
	public void cat(String remoteFile) throws IOException {
		Path path = new Path(remoteFile);
		FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf);
		FSDataInputStream fsdis = null;
		System.out.println("cat: " + remoteFile);
		try {
			fsdis = fs.open(path);
			IOUtils.copyBytes(fsdis, System.out, 4096, false);
		} finally {
			IOUtils.closeStream(fsdis);
			fs.close();
		}
	}
}

发表于 | 分类于 |

环境

1
2
3
4
ubuntu 16.04
4台机器的Hadoop2.7.2集群
Mysql安装在slave2中
hive安装在master上

下载Hive

1
2
3
$ wget http://mirrors.cnnic.cn/apache/hive/hive-1.2.1/apache-hive-1.2.1-bin.tar.gz
# 解压
$ tar -zxvf apache-hive-1.2.1-bin.tar.gz /home/ubuntu/cloud

配置Hive环境变量

1
2
3
4
5
6
7
$ sudo vim /etc/profile
#添加
export HIVE_HOME=/home/ubuntu/cloud/apache-hive-1.2.1-bin
export PATH=$PATH:$HIVE_HOME/bin
$source /etc/profile

在Mysql中创建Hive用户

1
2
3
mysql>CREATE USER 'hive' IDENTIFIED BY 'hive';
mysql>GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' IDENTIFIED BY 'hive' WITH GRANT OPTION;
mysql>flush privileges;

创建Hive数据库

1
2
$ mysql -uhive -phive
mysql>create database hive;

配置Hive

进入Hive的conf目录,找到hive-default.xml.template，cp份为hive-site.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ vim hive-site.xml
# 删除configuration标签里的所有内容 添加如下内容
	<property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://slave2:3306/hive?createDatabaseIfNotExist=true</value>
        <description>JDBC connect string for a JDBC metastore</description>    
	</property>   
	<property> 
        <name>javax.jdo.option.ConnectionDriverName</name> 
        <value>com.mysql.jdbc.Driver</value> 
        <description>Driver class name for a JDBC metastore</description>     
	</property>               
 
	<property> 
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>hive</value>
        <description>username to use against metastore database</description>
	</property>
	<property>  
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>hive</value>
        <description>password to use against metastore database</description>  
	</property>

下载mysql-connector-java-5.1.32-bin.jar

这里用5.1.32版本测试不报错，5.1.38会报warn

1
2
#将连接jar包拷贝到Hive的lib目录
$ cp mysql-connector-java-5.1.32-bin.jar /home/ubuntu/cloud/apache-hive-1.2.1-bin/lib/

若要装hive客户端可在客户端节点设置

vim hive-site.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<configuration>
<!-- thrift://<host_name>:<port> 默认端口是9083 -->
<property>
<name>hive.metastore.uris</name>
  <value>thrift://master:9083</value>
  <description>Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.</description>
</property>
 
<!--  hive表的默认存储路径 -->
<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>/user/hive/warehouse</value>
  <description>location of default database for the warehouse</description>
</property>
</configuration>

Hive启动

要启动metastore服务

1
2
3
4
5
6
7
8
9
$ hive --service metastore &
$ jps
10288 RunJar  #多了一个进程
9365 NameNode
9670 SecondaryNameNode
11096 Jps
9944 NodeManager
9838 ResourceManager
9471 DataNode

启动hive命令行

1
2
3
4
5
6
ubuntu@master:~$ hive
Logging initialized using configuration in jar:file:/home/ubuntu/cloud/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar!/hive-log4j.properties
hive> show tables;
OK
Time taken: 0.705 seconds

启动hiveserver2

1
hive --service hiveserver2 start &

问题解决

问题：创建表出先如下错误，删除表卡住

1
Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct MetaStore DB connections, we don't support retries at the client level.

初始化
注意：初始化之前先删除hdfs上的metastore,否则会出错/user/hive/warehouse
在hive服务端输入以下命令

1
schematool -dbType mysql -initSchema