Another breakpad server. Part 1

In the last quarter, they did MVP of the kresh service. An analogue of Socorro from Mozilla, but taking into account their requirements. The service code will be uploaded to GitHub as refactoring occurs. The utilities discussed in this article are available here .

We had the following requirements:

• getting report from Windows, Mac OS X, GNU / Linux;
• receiving a crash report from the web (collected via emscripten);
• collection of equipment data (CPU, GPU, Memory);
• grouping crashes by version, platform, user , reason;
• the application keeps logs, you need to store and log along with the report.

Content:

• Breakpad: symbol files and crash reports;
• Emscripten: compilation options, symbol files, error handling;
• Ui.

Breakpad

Breakpad has a utility that extracts character files from elf / pdb. Here is the file format description. This is a text file, but we are interested in the first line has the format MODULE operatingsystem architecture id name , here it looks like this:

 MODULE windows x86 9E8FC13F1B3F448B89FF7C940AC054A21 IQ Option.pdb MODULE Linux x86_64 4FC3EB040E16C7C75481BC5AA03EC8F50 IQOption MODULE mac x86_64 B25BF49C9270383E8DE34560730689030 IQOption 

Further, these files should be placed in a special order: base_dir/name/id/name.sym , it looks like this:

 base_dir/IQ Option/9E8FC13F1B3F448B89FF7C940AC054A21/IQ Option.sym base_dir/IQOption/4FC3EB040E16C7C75481BC5AA03EC8F50/IQOption.sym base_dir/IQOption/B25BF49C9270383E8DE34560730689030/IQOption.sym 

To get a crash report, you can use the minidump_stackwalk utility from the breakpad distribution:

 $ minidump_stackwalk path_to_crash base_dir 

This utility can output both in human readable form and in machine-readable format.

But this is not very convenient. The Mozilla Socorro includes a stackwalker utility that issues json (example on crash-stats.mozilla.com )

Emscripten

Catching falls is possible through the global window.onerror handler. Depending on the browser, the messages will differ:

 Uncaught abort() at Error at jsStackTrace (http://...) at Object.abort (http://...) at _abort (http://...) at _free (...) at __ZN2F28ViewMain13setFullscreenEb (...) at Array.__ZNSt3__210__function6__funcIZN2F28ViewMainC1EvE4__13NS_9allocatorIS4_EEFbPNS2_9UIElementEEEclEOS8_ (http://...) at __ZNKSt3__28functionIFllEEclEl (http://...) at __ZNK2F26detail23multicast_function_baseIFbPNS_9UIElementEENS_24multicast_result_reducerIFbRKNSt3__26vectorIbNS6_9allocatorIbEEEEEXadL_ZNS_10atLeastOneESC_EEEEiLin1EEclERKS3_ (http://...) at __ZN2F29UIElement14processTouchUpERKNS_6vec2_tIfEE (http://...) If this abort() is unexpected, build with -s ASSERTIONS=1 which can give more information. 

 uncaught exception: abort() at jsStackTrace@http://localhost/traderoom/glengine.js?v=1485951440.84:1258:13 stackTrace@http://... abort@http://... _abort@http://... _free@http://... __ZN2F28ViewMain13setFullscreenEb@http://... __ZNSt3__210__function6__funcIZN2F28ViewMainC1EvE4__13NS_9allocatorIS4_EEFbPNS2_9UIElementEEEclEOS8_@http://... __ZNKSt3__28functionIFllEEclEl@http://... __ZNK2F26detail23multicast_function_baseIFbPNS_9UIElementEENS_24multicast_result_reducerIFbRKNSt3__26vectorIbNS6_9allocatorIbEEEEEXadL_ZNS_10atLeastOneESC_EEEEiLin1EEclERKS3_@http://... __ZN2F29UIElement14processTouchUpERKNS_6vec2_tIfEE@http://... __ZN2F29UIElement17processTouchEventERNSt3__26vectorINS1_4pairIPS0_NS_6vec2_tIfEEEENS1_9allocatorIS7_EEEEjNS_15UI_TOUCH_ACTIONE@http://... __ZN2F29UIElement5touchEffNS_15UI_TOUCH_ACTIONEj@http://... __ZN2F213MVApplication5touchEffNS_15UI_TOUCH_ACTIONEj@http://... at stackTrace (http://... glengine.js?v=1485951440.84:360629:11 __ZN2F27UIInput7processEj@http://... __Z14on_mouse_eventiPK20EmscriptenMouseEventPv@http://... dynCall_iiii@http://... dynCall@http://... handlerFunc@http://... jsEventHandler@http://... If this abort() is unexpected, build with -s ASSERTIONS=1 which can give more information. 

Such a message is created if you use the -g switch when compiling. On our project, the size of the output code asm.js is 3 times larger. Therefore, we use - emit-symbol-map .
At the output, we get a file with characters in a simple key: value format:

 $cc:__ZNSt3__210__function6__funcIZN2F28ViewMain17animateLeftPannelEbE4__36NS_9allocatorIS4_EEFvvEEclEv f8d:__ZNKSt3__210__function6__funcIZN2F218MVMessageQueueImpl4sendINS2_26EventSocialProfileReceivedEJiEEEvDpRKT0_EUlvE_NS_9allocatorISA_EEFvvEE7__cloneEPNS0_6__baseISD_EE Z1:__ZN2F211recognizers24UIPinchGestureRecognizer6updateEPNS_9UIElementERKNS_6vec2_tIfEEj 

and the messages now look like:

 Uncaught abort() at Error at jsStackTrace (http://...) at stackTrace (http://...) at Object.abort (http://...) at _abort (http://...) at Eb (http://...) at Xc (http://...) at rc (http://...) at Array.$c (http://...) at Pc (http://...) at Array.Wb (http://...) If this abort() is unexpected, build with -s ASSERTIONS=1 which can give more information. 

To get a call stack, an auxiliary utility was written:

 #include <map> #include <set> #include <list> #include <regex> #include <fstream> #include <iostream> #include <cxxabi.h> #include <rapidjson/writer.h> #include <rapidjson/stringbuffer.h> namespace { struct Deleter { void operator()(char *data) const { free((void *) data); } }; using CharPtr = std::unique_ptr<char, Deleter>; } std::string demangle(const std::string &mangledName) { int status = 0; int shift = 0; if (mangledName[1] == '_') { shift = 1; } CharPtr realname(abi::__cxa_demangle(mangledName.data() + shift, 0, 0, &status)); if (status == 0) { return std::string(realname.get()); } else { if (mangledName[0] == '_') { const auto str = mangledName.substr(1, mangledName.size() - 1); int status = 0; CharPtr realname(abi::__cxa_demangle(str.data(), 0, 0, &status)); if (status == 0) { return std::string(realname.get()); } return mangledName; } return mangledName; } } void printUsage() { std::cout << "webstackwalker crash_dump symbol_file" << std::endl; } std::map<std::string, std::string> SYMBOL_MAP; void readSymbols(const std::string &path); void flushUnParseLine(const std::string &line); int main(int argc, char **argv) { if (argc < 2) { printUsage(); return 1; } readSymbols(std::string(argv[2])); const std::string inputFile(argv[1]); std::ifstream input(inputFile); const std::regex re("^(?:\\s{4}at\\s){0,1}(?:Array\\.){0,1}([\\w\\d\\$]+)(?: \\(|@).+\\){0,1}"); rapidjson::StringBuffer buffer; rapidjson::Writer<rapidjson::StringBuffer> writer(buffer); writer.StartArray(); const std::set<std::string> skip = {"jsStackTrace", "stackTrace", "abort"}; while (!input.eof()) { std::smatch match; std::string line; std::getline(input, line); if (std::regex_search(line, match, re)) { if (skip.count(match[1])) { continue; } auto iter = SYMBOL_MAP.find(match[1]); std::string function; if (iter != SYMBOL_MAP.cend()) { function = demangle(iter->second); } else { function = demangle(match[1]); } writer.String(function.c_str()); } } writer.EndArray(); std::cout << buffer.GetString() << std::endl; return 0; } void readSymbols(const std::string &path) { std::ifstream input(path); if (!input.is_open()) { std::cerr << "Can't open symbols file: " << path << std::endl; exit(2); } const std::regex re("^([\\d\\w$]+):([\\d\\w]+)$"); while (!input.eof()) { std::smatch match; std::string line; std::getline(input, line); if (std::regex_search(line, match, re)) { SYMBOL_MAP[match[1]] = match[2]; } } } 

The utility uses demangle to convert:

 _ZN2F211recognizers24UIPinchGestureRecognizer6updateEPNS_9UIElementERKNS_6vec2_tIfEEj 

at

 F2::recognizers::UIPinchGestureRecognizer::update(F2::UIElement*, F2::vec2_t<float> const&, unsigned int) 

Ui

We add drop reports in Elasticsearch, so for the first time we use Kibana as a means of visualizing and analyzing the contents of the elastic.

When using kibana we get out of the box:

• dashboards with automatic updates;
• visualization:
  
  • grigs and charts;
  • tables.

Dashboards group crashes by platform, build, signature. The filter system allows you to find out:

• in which builds this bug occurs;
• on which platforms is played.

Applied filters can be transferred to the discover tab, where you can see the details of the fall. As it turned out the kibana has a modular structure that allows you to expand its capabilities. A simple plugin was written to add a report render, which is much more convenient than the standard Table and JSon.