8 min read

GDT_PyR 12 - Web services

Code R complet: https://forgemia.inra.fr/gdtpyr/gdt_pyr/-/tree/main/GDT_PyR12_Web_services

Generalities

Definition of web services

A Web service is a server running on a computer device, listening for requests at a particular port over a network, serving web documents (HTML, JSON, XML, Images), and creating web applications services, which serve in solving specific domain problems over the web (www, internet, HTTP)

https://en.wikipedia.org/wiki/Web_service

API (Application Programming Interface)

This is a gateway to communicate with an application. This is most of the time used to access data stored in the server, but it can also be required to send data to the web service, or request some functions (apply a prediction model, …). This is the entry point to an application.

A web service is a sub-part of APIs.

The request are made using the HTTP application protocol.

A web service is usually a layer of abstraction that disconnect the user from the technical architecture of the data (what kind of database, what language, which location). The only requirement if to provide the correct input/output format. This format is often the JSON format. That involve parsing from and to JSON to communicate with a web service.

In REST Web services. The actions are generally identified by a verb among GET, POST, UPDATE, DELETE, that follows the CRUD (Create, Read, Update, Delete) principles. Update and Delete are generally reserved for admin rights.

Little summary :

  • API
    • Web Services
      • REST API webservices

Public APIs

You can find a list of state APIs on the following website : https://api.gouv.fr/

For example :

Usualy a lot of commercial applications provide some kind of API to be included in the global scanners (Air flights, transports, …), as well as a lot of public informations such as geography, weather, …

Authentication

An important difference between APIs concerns the authentication rights. Depending on the privacy of the data handled, the API is more or less open. It can vary from a fully open API with no authentication to a private API with a strong privacy handled by cryptographic keys (tokens).
Usually an identification by username and password coupled by the presence of groups deals with the privacy issues.

Documentation of the web service

An API is a multitude of several services, which all have a dedicated action. In order to explore and request the appropriate service. You need a proper documentation.
Unfortunately all documentations are not following the same rules making the first steps more or less feasable. The public transport of Brest proposed a human readable documentation here for example. There is also a machine-readable documentation propoosed by Open-API and the software swagger.

OpenAPI - Swagger

OpenAPI specifications can be written in YAML or JSON. It allows you to describe your entire API, including:

  • Available endpoints (/users) and operations on each endpoint (GET /users, POST /users)
  • Operation parameters Input and output for each operation
  • Authentication methods
  • Contact information, license, terms of use and other information.

Swagger is a software that became a standard in documentation of RESTful API as well as automated code generator and tests. The autogeneration of documentation and client code in different language is a real plus. It helps to keep up to date with the code of the web services. The client code can be generated in R or Python as well as a multitude of other languages (TypeScript, …).


Interactions web services and R

The two packages that you will need to deal with web services in R are jsonlite to manipulate JSON format. And httr to perform HTTP request. in addition some encryption functions are desired and found in openssl package.

library(jsonlite)
library(httr)
library(openssl)
library(reticulate)  #include the Python language

A request is performed quite simply using the GET function. The difficult part is to find the correct web service to request and provide the correct parameters. The different parameters are included in the finalURL of the web service as a HTTP request : http://example.com/api?year=2014.
You can decompose an URL like this :

  • schema : HTTP (mailto/soap/ldap, …)
  • domain : example.com
  • path : /api
  • request : ?year=2014

Example of Brest transport

#The address of the webservice
brestStopsNames = "https://applications002.brest-metropole.fr/WIPOD01/Transport/REST/getStopsNames?format=JSON"
StopsNamesjson = httr::GET(url = brestStopsNames, encode="json")
# transform http response object into a JSON object and finally into a R object
StopsNames = jsonlite::fromJSON(httr::content(StopsNamesjson,  as = "text",  encoding = "UTF-8"))
head(StopsNames)
brestRoutes = "https://applications002.brest-metropole.fr/WIPOD01/Transport/REST/getRoutes?format=%7bformat%7d"
Routesjson = httr::GET(url = brestRoutes , encode="json")
Routes = jsonlite::fromJSON(httr::content(Routesjson,  as = "text",  encoding = "UTF-8"))
head(Routes)

The arguments are put in the URL as a request (following the ?). The difficulty is to write correctly this URL to perform the appropriate request. A R friendly approach is to create a list with the various arguments and concatenate the list with the base URL. This is the first step to create a function! To know the arguments you should read the documentation.

#https://applications002.brest-metropole.fr/WIPOD01/Transport/REST/getStop?format=json&stop_name=malakoff
# The parameters of the 'function'
baseBrest = "https://applications002.brest-metropole.fr/WIPOD01/Transport/REST"
serviceBrest = "/getStop?"
attriBrest = list(
  format = "json",
  stop_name = "malakoff"
)

# the process of the 'function'
URLBrest = paste(baseBrest, serviceBrest, sep = "")
urlParamsBrest = ""
for (attribut in names(attriBrest)) {
  if (urlParamsBrest != ""){
    urlParamsBrest = paste0(urlParamsBrest, "&")
  }
   #   character arguments
  if (is.character(attriBrest[[attribut]])){
    urlParamsBrest = paste0(urlParamsBrest, attribut, "=", utils::URLencode(attriBrest[[attribut]], reserved = TRUE))
    #   numeric arguments
  } else if (is.numeric(attriBrest[[attribut]])){
    urlParamsBrest = paste0(urlParamsBrest, attribut, "=", format(attriBrest[[attribut]], scientific = FALSE))
    #   other arguments
  } else {
    urlParamsBrest = paste0(urlParamsBrest, attribut, "=", attriBrest[[attribut]])
  }
}
BrestRequest = paste(URLBrest, urlParamsBrest, sep = "", collapse = "")

# The output of the 'function'
Stopjson = httr::GET(url = BrestRequest, encode="json")
Stops = jsonlite::fromJSON(httr::content(Stopjson,  as = "text",  encoding = "UTF-8"))
head(Stops)

Example of OpenSILEX PHIS

The documentation of the PHIS web services is generated through the swagger software, it allows you to perform some requests online.

wsurlTOKEN = "http://www.opensilex.org:8080/openSilexSandBoxAPI/rest/brapi/v1/token"
body = list(
  grant_type = "password",
  username = "admin@opensilex.org",
  password = toString(md5("admin"))
)
tokenjson = httr::POST(url = wsurlTOKEN, body = body, encode="json")
authorization = jsonlite::fromJSON(httr::content(tokenjson,  as = "text",  encoding = "UTF-8"))
print(tokenjson)
print(authorization)

# alternative
# raw = toJSON(finalbody, auto_unbox = TRUE)
# authorization = httr::POST(url = wsurlTOKEN, body = raw, encode="raw", content_type_json()) #le content_type_json est indispensable

Remarque : GET response will send the token in the URL (not secured) and POST response will send the token in the body (more secured) but still in plain text. authentication services should use httpS for encryption.

Collecting data - GET

WSurlGET = "http://www.opensilex.org:8080/openSilexSandBoxAPI/rest/experiments?pageSize=20&page=0"
authorizationToken = authorization$access_token
ExperimentJSON = httr::GET(WSurlGET, config = httr::add_headers(Authorization = paste("Bearer ", authorizationToken, sep = "")))
Experiment = jsonlite::fromJSON(httr::content(ExperimentJSON,  as = "text",  encoding = "UTF-8"))
Experiment

Note here the presence of headers carrying the authentication token.

Sending data - POST

To send data to the web services, the POST function is used. Same as the GET you should find the correct URL and arguments. The difference is the addition of a body argument that cast the request to the Web Service with the appropriate format.

WSurlPOST = "http://www.opensilex.org:8080/openSilexSandBoxAPI/rest/methods"
# you can send structured data to the web services, the big issue is translating it into a conveniant JSON format.
# services will also check the data you are sending (type, value, etc).
data = data.frame(
    label = "My_new_method",
    comment = "this is a test for G12_PyR!",
    ontologiesReferences =list(
        property = "http://www.w3.org/2004/02/skos/core#closeMatch",
        object = "http://www.cropontology.org/rdf/CO_715:0000139",
        seeAlso = "http://www.cropontology.org/ontology/CO_715/"
 )
)
 
dataJSON = httr::POST(WSurlPOST, config = httr::add_headers(Authorization=paste("Bearer ", authorizationToken, sep = "")), body = data, encode = "json")
data = jsonlite::fromJSON(httr::content(dataJSON,  as = "text",  encoding = "UTF-8"))
data
getMethod = "http://www.opensilex.org:8080/openSilexSandBoxAPI/rest/methods?pageSize=20&page=0&uri=http://www.opensilex.org/demo/id/methods/m012"
authorizationToken = authorization$access_token
methodJSON = httr::GET(getMethod, config = httr::add_headers(Authorization=paste("Bearer ", authorizationToken, sep = "")))
method = jsonlite::fromJSON(httr::content(methodJSON,  as = "text",  encoding = "UTF-8"))
method

This kind of request can obviously be formated into a package that helps you deal with web service calls as a function call. This client package can be provided by the API or generated automatically using the swagger code-generator tool. Or also be written by hand if you are confident enough. (see for example the package to access the web Services of PHIS here )

This aproach is efficient but sensitive to any update on the web service. Strong advantage of auto-generated packages.

Interactions web services and Python

from pprint import pprint

import requests, json, hashlib

host = 'http://www.opensilex.org:8080/openSilexSandBoxAPI/rest/'

headers = { 
    'Content-Type': 'application/json',
    'accept' : 'application/json',
}

################ Authentification with a POST request ################

data = {
    'grant_type': 'password',
    'username': 'admin@opensilex.org',
    'password': hashlib.md5('admin'.encode('utf-8')).hexdigest(),
    'client_id': 'string'
}

json_data = json.dumps(data).encode('utf-8')

url = host + 'brapi/v1/token'
response = requests.post(url, headers = headers, data = json_data)
print(response.text + "\n")

token = response.json()['access_token'];
print(token)

############################# GET request ############################

host = 'http://www.opensilex.org:8080/openSilexSandBoxAPI/rest/'

headersMetadata = { 
    'Content-Type': 'application/json',
    'accept' : 'application/json',
    'Authorization':'Bearer ' + token
}

variableUri = 'http://www.opensilex.org/demo/id/variables/v001'


headers = { 'Content-Type': 'application/json',
'accept' : 'application/json',
'Authorization':'Bearer ' + token
 }

serviceUrl = host + 'data?variable=' + variableUri

response = requests.get(serviceUrl, headers=headers)

pprint(response.text)

Transform your code into an API

A wink to Japan’s University of Tokyo ut-biomet :

https://github.com/ut-biomet/PlumberTutorial

Documentation

OpenSILEX PHIS package : https://github.com/OpenSILEX/phisWSClientR
Wikipedia : https://en.wikipedia.org/wiki/Web_service
API.gouv : https://api.gouv.fr/
API transport Brest : https://geo.pays-de-brest.fr/donnees/Documents/Public/DocWebServicesTransport.pdf
R-bloggers : https://www.r-bloggers.com/wrapping-access-to-web-services-in-r-functions/
OpenAPI : https://swagger.io/docs/specification/about/
Code HTTP : https://fr.wikipedia.org/wiki/Liste_des_codes_HTTP

sessionInfo()
## R version 4.1.2 (2021-11-01)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 10 x64 (build 19044)
## 
## Matrix products: default
## 
## locale:
## [1] LC_COLLATE=French_France.1252  LC_CTYPE=French_France.1252   
## [3] LC_MONETARY=French_France.1252 LC_NUMERIC=C                  
## [5] LC_TIME=French_France.1252    
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.7      bookdown_0.24   lattice_0.20-45 png_0.1-7      
##  [5] digest_0.6.29   grid_4.1.2      R6_2.5.1        jsonlite_1.8.0 
##  [9] magrittr_2.0.1  evaluate_0.14   blogdown_1.7    stringi_1.7.6  
## [13] rlang_0.4.12    jquerylib_0.1.4 Matrix_1.3-4    bslib_0.3.1    
## [17] reticulate_1.22 rmarkdown_2.11  tools_4.1.2     stringr_1.4.0  
## [21] xfun_0.29       yaml_2.2.1      fastmap_1.1.0   compiler_4.1.2 
## [25] htmltools_0.5.2 knitr_1.37      sass_0.4.0