Methodology for collecting and processing open data

Laboratory members are developing and actively using a methodology for collecting open data using the Requests, Selenium libraries in Python and API, as well as isolating attributes from the html pages. Problems are detected when collecting open data. Options for using alternative methods for collecting open data to bypass existing site and API restrictions are being considered.

The methodology for collecting and processing open data is used to implement research projects – for example, to analyze court open data or YouTube data. Youtube API data is used in computational social science research as part of quantitative content analysis to study the dynamics of the popularity of certain phenomena and the attitude of the Youtube audience to certain thematic content.

Another example of such work is the collection of data through the API service of the electronic library eLibrary, within which a methodology was developed for collecting, preprocessing and analyzing data from publication texts from the specified library, which formed the basis of the laboratory product – the computer program “Bib-eLib”.