Microservice on GO for grabbing tweet videos

Good afternoon, Habravchane! An article for beginners, some of the most new ideas you will not see here. Yes, and this functionality is likely to be implemented dozens of times in different languages. The idea is to get a link to the post on Twitter which contains the video, pick up this video and convert it to mkv.

To the point!

What we need:

• GO
• ffmpeg
• docker (although it can be without him. However, where without him these days?!;)

We have a link to a tweet from the video:

https://twitter.com/FunnyVines/status/1101196533830041600 

Of the entire link, we are only interested in the ID consisting of numbers, therefore, with an elementary regular schedule, we pull out the entire digital substring:

 var reurl = regexp.MustCompile(`\/(\d*)$`) //   -  e.GET("/*video", func(c *fasthttp.RequestCtx) { //      url := reurl.FindSubmatch([]byte(c.UserValue("video").(string))) 

With the received ID go to the address:

 resp, err := client.Get("https://twitter.com/i/videos/tweet/" + id) 

Where we get the link to the JS code of the video player:

src="https://abs.twimg.com/web-video-player/TwitterVideoPlayerIframe.f52b5b572446290e.js"
From this js file we need one very important thing - bearer for authorization in api twitter.

Regex'pim it!

 re, _ := regexp.Compile(`(?m)authorization:\"Bearer (.*)\",\"x-csrf`) 

This is not enough to access api, guest_token is still needed. You can get it by sending a POST request to the address - " https://api.twitter.com/1.1/guest/activate.json ", transferring there: personalization_id and guest_id from a cookie (which we received in response from the server when referring to the previous URL):

  var personalization_id, guest_id string cookies := resp.Cookies() for _, cookie := range cookies { if cookie.Name == "personalization_id" { personalization_id = cookie.Value } if cookie.Name == "guest_id" { guest_id = cookie.Value } } // // Get Activation url, _ := url.Parse("https://api.twitter.com/1.1/guest/activate.json") request := &http.Request{ Method: "POST", URL: url, Header: http.Header{ "user-agent": []string{"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"}, "accept-encoding": []string{"gzip", "deflate", "br"}, "authorization": []string{"Bearer " + bearer}, "cookie": []string{"personalization_id=\"" + personalization_id + "\"; guest_id=" + guest_id} }, } resp, err = client.Do(request) 

The answer is gzipped, in Go you can unpack it, like this:

 res, err := gzip.NewReader(resp.Body) if err != nil { return "", err } defer res.Close() r, err := ioutil.ReadAll(res) 

All right! Now we have everything we need to call the API:

  url, _ = url.Parse("https://api.twitter.com/1.1/videos/tweet/config/" + id + ".json") request = &http.Request{ Method: "GET", URL: url, Header: http.Header{ "user-agent": []string{"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"}, "accept-encoding": []string{"gzip", "deflate", "br"}, "origin": []string{"https://twitter.com"}, "x-guest-token": []string{gt.GuestToken}, "referer": []string{"https://twitter.com/i/videos/tweet/" + id}, "authorization": []string{"Bearer " + bearer}}, } resp, err = client.Do(request) 

The response from the API will be Json with the description of the video and, most importantly, the URL address for receiving it (playbackUrl):

 {"contentType":"media_entity","publisherId":"4888096512","contentId":"1096941371649347584","durationMs":11201,"playbackUrl":"https:\/\/video.twimg.com\/ext_tw_video\/1096941371649347584\/pu\/pl\/xcBvPmwAmKckck-F.m3u8?tag=6","playbackType" 

Finally, we have the address of the video, send it to ffmpeg, while checking in what video format, I saw 2 possible formats, the first one is mp4:

 if strings.Contains(videoURL.Track.PlaybackURL, ".mp4") { convert := exec.Command("ffmpeg", "-i", videoURL.Track.PlaybackURL, "-c", "copy", "./videos/"+id+".mkv") convert.Stdout = os.Stdout convert.Stderr = os.Stderr if convert.Run() != nil { return "", err } return id, nil } 

And the second - m3u8 playlist file, for this option you need one more step - GET'om get
it, and take the content URL in the desired resolution:

 #EXT-X-INDEPENDENT-SEGMENTS #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=256000,RESOLUTION=180x316,CODECS="mp4a.40.2,avc1.4d0015" /ext_tw_video/1039516210948333568/pu/pl/180x316/x0HWMgnbSJ9y6NFL.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=832000,RESOLUTION=464x816,CODECS="mp4a.40.2,avc1.4d001f" /ext_tw_video/1039516210948333568/pu/pl/464x816/Z58__ptq1xBk8CIV.m3u8 

And still - ffmpeg.

And now, a little about HTTP server

I used:

• fasthttp
• fasthttprouter
• tcplisten

The logic is as follows: we start the server:

 cfg := tcplisten.Config{ ReusePort: true, FastOpen: true, DeferAccept: true, Backlog: 1024, } ln, err := cfg.NewListener("tcp4", ":8080") if err != nil { log.Fatalf("error in reuseport listener: %s\n", err) } serv := fasthttp.Server{Handler: e.Handler, ReduceMemoryUsage: false, Name: "highload", Concurrency: 2 * 1024, DisableHeaderNamesNormalizing: true} if err := serv.Serve(ln); err != nil { log.Fatalf("error in fasthttp Server: %s", err) } 

And we process only one route / * video:

 //  : //  http://localhost:8080/https://twitter.com/FunnyVines/status/1101196533830041600 //   http://localhost:8080/1101196533830041600 e.GET("/*video", func(c *fasthttp.RequestCtx) { 

How can all this be collected?

For example, a simple Makefile (make build, make run ...):

 build: go build -o main docker build -t tvideo . run: go build -o main docker build -t tvideo . docker kill tvideo docker run -d --rm --name tvideo -v /etc/ssl:/etc/ssl:ro -v videos:/opt/videos -p 8080:8080 tvideo docker logs -f tvideo 

Pay attention to the "-v / etc / ssl: / etc / ssl: ro" flag, in the base image ubuntu there were no root certificates and the http client did not recognize https twitter, ran it from the host machine via --volume (now, like as, it is more correct to use --mount ).

Dockerfile

 FROM ubuntu //     docker image COPY main /opt/app RUN apt-get update && \ //   apt-get install -y ffmpeg && \ chmod +x /opt/app EXPOSE 8080 WORKDIR /opt CMD ./app 

Undoubtedly, I did not discover America in this article, but suddenly someone will come in handy.

Sources are available here .