java爬取并下载酷狗TOP500歌曲的方法

是这样的，之前买车送的垃圾记录仪不能用了，这两天狠心买了好点的记录仪，带导航、音乐、蓝牙、4g等功能，寻思，既然有这些功能就利用起来，用4g听歌有点奢侈，就准备去酷狗下点歌听，居然都是需要办会员才能下载，而且vip一月只能下载300首，我这么穷又这么抠怎么可能冲会员，于是百度搜了下怎么免费下载，都是python爬取，虽然也会一点，但是电脑上没安装python，再安装再研究感觉有点费劲，于是就花了半小时做了这个爬虫，技术一般，只记录分析实现过程，大牛请绕行。其中用到了一些库，包括：jsoup、httpclient、net.sf.json大家可以自行去下载jar包

1、分析是否能获得top500歌单

首先，打开酷狗首页查看酷狗top500，说好的500首，怎么就只有22首呢，

是真的只让看这些还是能找到其余的呢，于是我就看了下这top500的链接

https://www.kugou.com/yy/rank/home/1-8888.html?from=rank

可以看的出home后边有个1，难道这是代表第一页的意思？于是我就把1改成2，进入，果然进入了第二页，至此可以知道我们可以在网页里获取这500首的歌单。

2.分析找到真正的mp3下载地址（这个有点绕）

点一个歌曲进入播放页面，使用谷歌浏览器的控制台的elements，搜一下mp3，很轻松就定位到了mp3的位置，

但是使用java访问的时候爬取的html里却没有该mp3的文件地址，那么这肯定是在该页面的位置使用了js来加载mp3，那么刷新下网页，看网页加载了哪些东西，加载的东西有点多，着重看一下js、php的请求，主要是看里面有没有mp3的地址，分析细节就不用说了，

最终我在列表的

https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jquery191027067069941080546_1546235744250&hash=667939c6e784265d541deee65ae4f2f8&album_id=0&_=1546235744251

这个请求里发现了mp3的完整地址，

"play_url": "http://fs.w.kugou.com/201812311325/dcf5b6449160903c6ee48035e11434bb/g128/m08/02/09/iicbafrzqf2anoadadn94ubomau995.mp3",

那这个js是怎么判断是哪首歌的呢，那么只可能是hash这个参数来决定歌曲的，然后到播放页面里找到这个hash的位置，是在下面的js里

var datafromsmarty = [{"hash":"667939c6e784265d541deee65ae4f2f8","timelength":"237051","audio_name":"u767du5c0fu767d - u6700u7f8eu5a5au793c","author_name":"u767du5c0fu767d","song_name":"u6700u7f8eu5a5au793c","album_id":0}],//当前页面歌曲信息

playtype = "search_single";//当前播放

</script>

在去java爬取该网页，查看能否爬到这个hash，果然，爬取的html里有这段js，到现在mp3的地址也找到了，歌单也找到了，那么下一步就用程序实现就可以了。

3.java实现爬取酷狗mp3

先看一下爬取结果

找到了资源，程序实现就好说了，其中使用到了自己写的几个工具类，自己整理点自己的工具类还是有好处的，以后遇到什么问题就没必要重新写了，直接拿来用就可以了。没什么好说的了，下面直接贴出源码

spiderkugou.java

packagecom.bing.spider;

importjava.io.ioexception;

importjava.util.regex.matcher;

importjava.util.regex.pattern;

importorg.jsoup.nodes.document;

importorg.jsoup.nodes.element;

importorg.jsoup.select.elements;

importcom.bing.download.filedownload;

importcom.bing.html.htmlmanage;

importcom.bing.http.httpgetconnect;

importnet.sf.json.jsonobject;

publicclassspiderkugou{

publicstaticstringfilepath="f:/music/";

publicstaticstringmp3="https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jquery191027067069941080546_1546235744250&"

+"hash=hash&album_id=0&_=time";

publicstaticstringlink="https://www.kugou.com/yy/rank/home/page-8888.html?from=rank";

//"https://www.kugou.com/yy/rank/home/page-23784.html?from=rank";

publicstaticvoidmain(string[]args)throwsioexception{

for(inti=1;i<23;i++){

stringurl=link.replace("page",i+"");

gettitle(url);

//download("https://www.kugou.com/song/mfy6je5.html");

}

}

publicstaticstringgettitle(stringurl)throwsioexception{

httpgetconnectconnect=newhttpgetconnect();

stringcontent=connect.connect(url,"utf-8");

htmlmanagehtml=newhtmlmanage();

documentdoc=html.manage(content);

elementele=doc.getelementsbyclass("pc_temp_songlist").get(0);

elementseles=ele.getelementsbytag("li");

for(inti=0;i<eles.size();i++){

elementitem=eles.get(i);

stringtitle=item.attr("title").trim();

stringlink=item.getelementsbytag("a").first().attr("href");

download(link,title);

}

returnnull;

}

publicstaticstringdownload(stringurl,stringname)throwsioexception{

stringhash="";

httpgetconnectconnect=newhttpgetconnect();

stringcontent=connect.connect(url,"utf-8");

htmlmanagehtml=newhtmlmanage();

stringregex=""hash":"[0-9a-z]+"";

//编译正则表达式

patternpattern=pattern.compile(regex);

matchermatcher=pattern.matcher(content);

if(matcher.find()){

hash=matcher.group();

hash=hash.replace(""hash":"","");

hash=hash.replace(""","");

}

stringitem=mp3.replace("hash",hash);

item=item.replace("time",system.currenttimemillis()+"");

system.out.println(item);

stringmp=connect.connect(item,"utf-8");

mp=mp.substring(mp.indexof("(")+1,mp.length()-3);

jsonobjectjson=jsonobject.fromobject(mp);

stringplayurl=json.getjsonobject("data").getstring("play_url");

system.out.print(playurl+"==");

filedownloaddown=newfiledownload();

down.download(playurl,filepath+name+".mp3");

system.out.println(name+"下载完成");

returnplayurl;

}

}

httpgetconnect.java

100

101

102

103

104

package com.bing.http;

import java.io.bufferedreader;

import java.io.ioexception;

import java.io.inputstream;

import java.io.inputstreamreader;

import java.security.nosuchalgorithmexception;

import java.security.cert.certificateexception;

import java.security.cert.x509certificate;

import javax.net.ssl.sslcontext;

import javax.net.ssl.trustmanager;

import javax.net.ssl.x509trustmanager;

import org.apache.commons.logging.log;

import org.apache.commons.logging.logfactory;

import org.apache.http.httpentity;

import org.apache.http.client.clientprotocolexception;

import org.apache.http.client.httpclient;

import org.apache.http.client.responsehandler;

import org.apache.http.client.config.requestconfig;

import org.apache.http.client.methods.closeablehttpresponse;

import org.apache.http.client.methods.httpget;

import org.apache.http.conn.clientconnectionmanager;

import org.apache.http.conn.scheme.scheme;

import org.apache.http.conn.scheme.schemeregistry;

import org.apache.http.conn.ssl.sslsocketfactory;

import org.apache.http.impl.client.basicresponsehandler;

import org.apache.http.impl.client.closeablehttpclient;

import org.apache.http.impl.client.defaulthttpclient;

import org.apache.http.impl.client.httpclients;

import org.apache.http.impl.conn.basichttpclientconnectionmanager;

import org.apache.http.params.httpparams;

/**

* @说明：

* @author: gaoll

* @createtime:2014-11-13

* @modifytime:2014-11-13

*/

public class httpgetconnect {

/**

* 获取html内容

* @param url

* @param charsetname utf-8、gb2312

* @return

* @throws ioexception

*/

public static string connect(string url,string charsetname) throws ioexception{

basichttpclientconnectionmanager connmanager = new basichttpclientconnectionmanager();

closeablehttpclient httpclient = httpclients.custom()

.setconnectionmanager(connmanager)

.build();

string content = "";

try{

httpget httpget = new httpget(url);

requestconfig requestconfig = requestconfig.custom()

.setsockettimeout(5000)

.setconnecttimeout(50000)

.setconnectionrequesttimeout(50000)

.build();

httpget.setconfig(requestconfig);

httpget.setheader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");

httpget.setheader("accept-encoding", "gzip,deflate,sdch");

httpget.setheader("accept-language", "zh-cn,zh;q=0.8");

httpget.setheader("connection", "keep-alive");

httpget.setheader("upgrade-insecure-requests", "1");

httpget.setheader("user-agent", "mozilla/5.0 (windows nt 6.1; wow64) applewebkit/537.36 (khtml, like gecko) chrome/45.0.2454.101 safari/537.36");

//httpget.setheader("hosts", "www.oschina.net");

httpget.setheader("cache-control", "max-age=0");

closeablehttpresponse response = httpclient.execute(httpget);

int status = response.getstatusline().getstatuscode();

if (status >= 200 && status < 300) {

httpentity entity = response.getentity();

inputstream instream = entity.getcontent();

bufferedreader br = new bufferedreader(new inputstreamreader(instream,charsetname));

stringbuffer sbf = new stringbuffer();

string line = null;

while ((line = br.readline()) != null){

sbf.append(line + " ");

}

br.close();

content = sbf.tostring();

} else {

content = "";

}

}catch(exception e){

e.printstacktrace();

}finally{

httpclient.close();

}

//log.info("content is " + content);

return content;

}

private static log log = logfactory.getlog(httpgetconnect.class);

}

htmlmanage.java

package com.bing.html;

import java.io.ioexception;

import java.util.arraylist;

import java.util.list;

import org.apache.commons.logging.log;

import org.apache.commons.logging.logfactory;

import org.jsoup.jsoup;

import org.jsoup.nodes.document;

import org.jsoup.nodes.element;

import org.jsoup.select.elements;

import com.bing.http.httpgetconnect;

/**

* @说明：

* @author: gaoll

* @createtime:2014-11-13

* @modifytime:2014-11-13

*/

public class htmlmanage {

public document manage(string html){

document doc = jsoup.parse(html);

return doc;

}

public document managedirect(string url) throws ioexception{

document doc = jsoup.connect( url ).get();

return doc;

}

public list<string> managehtmltag(document doc,string tag ){

list<string> list = new arraylist<string>();

elements elements = doc.getelementsbytag(tag);

for(int i = 0; i < elements.size() ; i++){

string str = elements.get(i).html();

list.add(str);

}

return list;

}

public list<string> managehtmlclass(document doc,string clas ){

list<string> list = new arraylist<string>();

elements elements = doc.getelementsbyclass(clas);

for(int i = 0; i < elements.size() ; i++){

string str = elements.get(i).html();

list.add(str);

}

return list;

}

public list<string> managehtmlkey(document doc,string key,string value ){

list<string> list = new arraylist<string>();

elements elements = doc.getelementsbyattributevalue(key, value);

for(int i = 0; i < elements.size() ; i++){

string str = elements.get(i).html();

list.add(str);

}

return list;

}

private static log log = logfactory.getlog(htmlmanage.class);

}

filedownload.java

package com.bing.download;

import java.io.bufferedinputstream;

import java.io.bufferedoutputstream;

import java.io.file;

import java.io.fileoutputstream;

import org.apache.commons.logging.log;

import org.apache.commons.logging.logfactory;

import org.apache.http.client.config.requestconfig;

import org.apache.http.client.methods.closeablehttpresponse;

import org.apache.http.client.methods.httpget;

import org.apache.http.impl.client.closeablehttpclient;

import org.apache.http.impl.client.httpclients;

/**

* @说明：

* @author: gaoll

* @createtime:2014-11-20

* @modifytime:2014-11-20

*/

public class filedownload {

/**

* 文件下载

* @param url 链接地址

* @param path 要保存的路径及文件名

* @return

*/

public static boolean download(string url,string path){

boolean flag = false;

closeablehttpclient httpclient = httpclients.createdefault();

requestconfig requestconfig = requestconfig.custom().setsockettimeout(2000)

.setconnecttimeout(2000).build();

httpget get = new httpget(url);

get.setconfig(requestconfig);

bufferedinputstream in = null;

bufferedoutputstream out = null;

try{

for(int i=0;i<3;i++){

closeablehttpresponse result = httpclient.execute(get);

system.out.println(result.getstatusline());

if(result.getstatusline().getstatuscode() == 200){

in = new bufferedinputstream(result.getentity().getcontent());

file file = new file(path);

out = new bufferedoutputstream(new fileoutputstream(file));

byte[] buffer = new byte[1024];

int len = -1;

while((len = in.read(buffer,0,1024)) > -1){

out.write(buffer,0,len);

}

flag = true;

break;

}else if(result.getstatusline().getstatuscode() == 500){

continue ;

}

}catch(exception e){

e.printstacktrace();

flag = false;

}finally{

get.releaseconnection();

try{

if(in != null){

in.close();

}

if(out != null){

out.close();

}

}catch(exception e){

e.printstacktrace();

flag = false;

}

return flag;

}

private static log log = logfactory.getlog(filedownload.class);

}

到这就结束了，有可能有些代码没贴全，主要代码已经差不多，应该可以跑起来，多多指教。

以上就是本文的全部内容，希望对大家的学习有所帮助，也希望大家多多支持快网idc。

原文链接：https://my.oschina.net/gllfeixiang/blog/2995570

java爬取并下载酷狗TOP500歌曲的方法

相关文章

快网idc优惠网

相关文章

微信

快网idc优惠网

QQ交流群