java_爬虫_从腾讯视频播放界面爬取视频真实地址
由于想在微信公众号里爬一点儿考研的视频
花了差不多一天的时间把这个爬虫做好(其实也不算爬虫吧,就算个能批量处理的地址解析器,半个爬虫)
不多说,进正题
(本文适合有java基础的同学,没基础的用客户端缓存然后格式转换吧)
所需条件:
1.一台联网的有java环境的电脑
2.耐心
访问后台接口网址:
http://vv.video.qq.com/getinfo(低清的只要这一个就好了)
http://vv.video.qq.com/getkey(高清的需要访问这个)
原理(获取低清视频,先把原理打通,高清后期有时间会更):
步骤一:
获取你想要下载的视频的腾讯视频页面地址(这个很容易啦,就不赘述)
此处以:https://v.qq.com/x/page/f08302y6rof.html为例
步骤二:
获取视频vid
此处的vid为f08302y6rof,就是上边儿网址上那一串长长的东西
步骤三:
用获取到的视频的vid替换下面红色标明的vid(这一步是访问后台接口得到json报文)
http://vv.video.qq.com/getinfo?vids=f08302y6rof&platform=101001&charge=0&otype=json&defn=shd
然后访问
步骤四:
在页面返回浏览器的json报文中,找到fn 和 fvkey
我这边儿传回的报文如下
- QZOutputJson={"dltype":1,"exem":0,"fl":{"cnt":2,"fi":[{"id":100701,"name":"msd","lmt":0,"sb":1,"cname":"标清;(270P)","br":29,"profile":2,"drm":0,"video":1,"audio":1,"fs":101567331,"super":0,"hdr10enh":0,"sl":1},{"id":2,"name":"mp4","lmt":0,"sb":1,"cname":"高清;(480P)","br":34,"profile":1,"drm":0,"video":1,"audio":1,"fs":130427092,"super":0,"hdr10enh":0,"sl":0}]},"hs":0,"ip":"111.79.225.65","ls":0,"preview":3383,"s":"o","sfl":{"cnt":0},"tm":1556431150,"vl":{"cnt":1,"vi":[{"br":29,"ch":0,"cl":{"fc":0,"keyid":"f08302y6rof.100701"},"ct":21600,"drm":0,"dsb":0,"fmd5":"74e3040ce70af50716abead16c9fba50","fn":"f08302y6rof.m701.mp4","fs":101567331,"fst":5,"fvkey":"D351DB69FA6EC791CB6DE47266F80B21BFFFAA3616A7B42975903ED5EA68589C0E2454137002A84799CF43B4FD972B415259C1F23C21CD34F2C34BC64F6D7D16F21BF3BF94F22B09491FC9D8C96CFFA3B3177345807F34EFDDAF94449E72FC3B8C55751EE9EADC5F","head":0,"hevc":0,"iflag":0,"level":0,"lnk":"f08302y6rof","logo":1,"mst":8,"pl":null,"share":1,"sp":0,"st":2,"tail":0,"td":"3383.47","ti":"2020考研数学寒假计划(第一次课)","tie":0,"type":3,"ul":{"ui":[{"url":"http://ugcws.video.gtimg.com/uwMROfz0r5zAoaQXGdGnC2dfhzlOR5XW60pRw41PvMP8tDlH/","vt":106,"dtc":0,"dt":2},{"url":"http://ugcydzd.qq.com/uwMROfz0r5zAoaQXGdGlC2dfhznfaJdqBNmJ_NLSRfZb0kLT/","vt":146,"dtc":0,"dt":2},{"url":"http://ugcsjy.qq.com/uwMROfz0r5zAoaQXGdGlK2dfhzmm-mdByiC0fycrmmUBpCVq/","vt":176,"dtc":0,"dt":2},{"url":"http://video.dispatch.tc.qq.com/uwMROfz0r5zAoaQXGdGlLGdfhzn3bYHHUWfJ-3lk8pLFnjzb/","vt":0,"dtc":0,"dt":2}]},"vh":480,"vid":"f08302y6rof","videotype":0,"vr":0,"vst":2,"vw":270,"wh":0.5625,"wl":{"wi":[{"id":19,"x":14,"y":14,"w":84,"h":27,"a":100,"md5":"dcc9dc5c478c4100ea2817c5e6020f26","url":"http://puui.qpic.cn/vcolumn_pic/0/logo_qing_xi_color_336_108.png/0","surl":"https://puui.qpic.cn/vcolumn_pic/0/logo_qing_xi_color_336_108.png/0"}]},"uptime":1548118095,"fvideo":0,"fvpint":0,"swhdcp":0}]}};
(传回来的就是一行,我直接用java代码解析了,手动找费眼睛)
步骤五:
利用获取到的fn和fvkey构建视频下载地址
此处构建的是:
http://ugcws.video.gtimg.com/f08302y6rof.m701.mp4?vkey=2E657DF01414A1F95E0B3CF7F187CEB84B3E439F5D0BA2D7F052967654DEFDE53292F0BE8BCD373FA0F269BA6BE5CC1AD5CC4AEE269AB0B1C72261815608260190B1D14D9B1820B0394DAB0C8DA1D8561F3B3455FBE5BA27D618C81A0A233256DDDAB6429E3A05FF
把获取到的fn替换前边儿一个短的标红内容
fvkey替换后边儿长的标红内容
这就是完整的视频下载地址了,可以用迅雷下载
完成
源码如下(有错误或者不规范请大佬指出,个人机器上可以运行):
- package catchVedio;
- import java.io.BufferedReader;
- import java.io.BufferedWriter;
- import java.io.File;
- import java.io.FileReader;
- import java.io.FileWriter;
- import java.io.IOException;
- import java.io.InputStreamReader;
- import java.io.UnsupportedEncodingException;
- import java.net.HttpURLConnection;
- import java.net.URL;
- import java.util.ArrayList;
- import java.util.List;
- /**
- * 获取视频接口的json
- * @author Administrator
- *
- */
- public class CatchVedio {
- // Socket client = new Scoket();
- private URL url;
- private HttpURLConnection urlConnection;
- private int responseCode;
- private BufferedReader reader;
- private BufferedWriter writer;
- public static void main(String[] args) {
- CatchVedio cv = new CatchVedio();
- try {
- String[] VedioURL = cv.get_VedioURL();//接收
- for(String temp:VedioURL) {//temp是每一个视频的播放地址
- cv.toDownloadURL(cv.analyse(cv.get_Json(temp)));//写出到文件
- }
- } catch (IOException e) {
- // TODO 自动生成的 catch 块
- e.printStackTrace();
- }finally {
- try {
- cv.reader.close();
- cv.writer.close();
- } catch (IOException e) {
- // TODO 自动生成的 catch 块
- e.printStackTrace();
- }
- }
- }
- void toDownloadURL(String real_url) throws IOException {//将对应下载地址输出到文件
- this.writer = new BufferedWriter(new FileWriter("D:/worm/downloadURL.txt",true));//定义追加方式写入的流
- // this.writer.append(real_url);
- this.writer.write(real_url+"\r\n");
- this.writer.flush();
- }
- String analyse(String json) {//分析json,传回完整下载地址
- int fvkey_index = json.indexOf("\"fvkey\":\"")+9;
- int endIndex = json.indexOf("\"",fvkey_index);
- String fvkey = json.substring(fvkey_index,endIndex);//获取到fvkey
- // System.out.println(fvkey);
- int fn_index = json.indexOf("\"fn\":\"")+6;
- int fn_end = json.indexOf("\"",fn_index);
- String fn = json.substring(fn_index,fn_end);//获取到视频文件名
- // System.out.println(fn);
- String head = "http://ugcws.video.gtimg.com/";
- StringBuffer real_url = new StringBuffer();
- real_url.append(head);//加入头部
- real_url.append(fn+"?");//加入文件名
- real_url.append("vkey="+fvkey);//加入解锁码
- /*构造成功*/
- // System.out.println(real_url.toString());
- return real_url.toString();
- }
- String get_Json(String url) throws UnsupportedEncodingException, IOException {
- String line = "";
- StringBuffer sb = new StringBuffer();
- this.url = new URL(url);
- this.urlConnection = (HttpURLConnection)this.url.openConnection();
- this.responseCode = this.urlConnection.getResponseCode();
- if (this.responseCode == 200) {
- this.reader = new BufferedReader(new InputStreamReader(this.urlConnection.getInputStream(), "UTF-8"));
- while ((line = this.reader.readLine()) != null) {
- sb.append(line);// 网页传回的只有一行
- }
- return sb.toString();
- }
- return "";
- }
- String[] get_VedioURL() throws IOException {
- // void get_VedioURL() throws IOException {
- File file = new File("D:/worm/vedioURL.txt");
- String line = "";
- this.reader = new BufferedReader(new FileReader(file));
- String[] t = new String[0];
- List<String> container = new ArrayList<String>();
- while(null!=(line = this.reader.readLine())) {
- if(line.equals("")) {
- continue;
- }
- line = this.change(line);//转换一下
- container.add(line);//装入容器
- }
- return container.toArray(t);
- }
- /**
- * http://vv.video.qq.com/getinfo?vids=x0164ytbgov&platform=101001&charge=0&otype=json&defn=shd //格式
- * @param str
- * @return
- * https://v.qq.com/x/page/f08302y6rof.html//页面地址示例
- * https://v.qq.com/x/page/y083158hphd.html
- * https://v.qq.com/x/page/c08503oe58c.html
- */
- String change(String str) {//定义从页面播放地址获取vid转换到后台接口地址的方法
- String head = "http://vv.video.qq.com/getinfo?vids=";
- String tail = "&platform=101001&charge=0&otype=json&defn=shd";
- String vid = str.substring(str.indexOf("page/")+5,str.indexOf(".html"));
- return head+vid+tail;
- }
- }
我是输入输出都是文件操作
希望对大家有所帮助
以上