Develpers Corner

Contents page

REST API

One way to access the Data Portal is to use the REST API. Most of the portal core functionalities are available through the application programming interface (API), which encompasses most of what you can do with the web interface. The information retrieved can then be used by an external code to transform, update or reference and provide new input for further calls to the API.

REST API for Open Data Portal(DKAN)

Open Data Portal is based on DKAN Open Data Platform, which includes a number of APIs to allow it to communicate with external applications. DKAN comes with three API’s:

  • the Dataset API,
  • the Datastore API,
  • the Dataset REST API.

These can be queried to discover metadata associated with the catalog or a specific dataset. It can also be harvested by other portals by complying with the API standard data.json.

Dataset REST API

The DKAN Dataset REST API uses the Services module to create CRUD endpoint at api/dataset/node. By default, this endpoint provides full CRUD access to a website’s content nodes, and limited access to users (to allow authentication). Most common operations on datasets and resources are retrieving, creating, updating and deleting specific node or adding a file to resource. Node is the most basic data structure in DKAN and can be access in /api/dataset/node/{nid}.

Retrieve Dataset

Request

GET http://docker:32774/api/dataset/node/75
Accept: application/json
X-CSRF-Token: XBWI44XD33XBIANLpyK-rtvRa0N5OcaC03qLx0VQsP4
Cookie: SESSd14344a17ca11d13bda8baf612c0efa5=OBoeXKMQx3zmaZrS_v3FOP7_Ze66fYA61TGhtm9s0Qk

Response

{
    "body": [],
    "changed": "1492544349",
    "comment": "0",
    "created": "1492544348",
    "data": "b:0;",
    "field_additional_info": [],
    "field_author": [],
    "field_conforms_to": [],
    "field_contact_email": [],
    "field_contact_name": [],
    "field_data_dictionary": [],
    "field_data_dictionary_type": [],
    "field_frequency": [],
    "field_granularity": [],
    "field_harvest_source_ref": [],
    "field_is_part_of": [],
    "field_landing_page": [],
    "field_language": [],
    "field_license": {
        "und": [
            {
                "format": null,
                "safe_value": "notspecified",
                "value": "notspecified"
            }
        ]
    },
    "field_modified_source_date": [],
    "field_orphan": {
        "und": [
            {
                "value": "0"
            }
        ]
    },
    "field_pod_theme": [],
    "field_public_access_level": {
        "und": [
            {
                "value": "public"
            }
        ]
    },
    "field_related_content": [],
    "field_resources": {
        "und": [
            {
                "target_id": "76"
            }
        ]
    },
    "field_rights": [],
    "field_spatial": [],
    "field_spatial_geographical_cover": [],
    "field_tags": [],
    "field_temporal_coverage": [],
    "field_topic": [],
    "language": "und",
    "log": "",
    "name": "admin",
    "nid": "75",
    "og_group_ref": [],
    "path": "http://docker:32774/dataset/test-dataset-16",
    "picture": "0",
    "promote": "0",
    "revision_timestamp": "1492544349",
    "revision_uid": "1",
    "status": "1",
    "sticky": "0",
    "title": "Test Dataset",
    "tnid": "0",
    "translate": "0",
    "type": "dataset",
    "uid": "1",
    "uuid": "d53881b3-d80f-49c2-8815-897321fe926e",
    "vid": "117",
    "vuuid": "c4663ada-0162-4780-8ee5-347c6c037429"
}

Create Dataset

Request

POST http://docker:32774/api/dataset/node
Content-Type: application/json
Accept: application/json
X-CSRF-Token: XBWI44XD33XBIANLpyK-rtvRa0N5OcaC03qLx0VQsP4
Cookie: SESSd14344a17ca11d13bda8baf612c0efa5=OBoeXKMQx3zmaZrS_v3FOP7_Ze66fYA61TGhtm9s0Qk
Content-Length: 44
{
    "type": "dataset",
   "title": "Test Dataset"
}

Response

{
    "nid": "75",
    "uri": "http://docker:32774/api/dataset/node/75"
}

Delete Dataset

Request

DELETE http://docker:32774/api/dataset/node/123
Content-Type: application/json
X-CSRF-Token: XBWI44XD33XBIANLpyK-rtvRa0N5OcaC03qLx0VQsP4

Update Dataset

Request

Content-Type: application/json
Accept: application/json
X-CSRF-Token: XBWI44XD33XBIANLpyK-rtvRa0N5OcaC03qLx0VQsP4
Cookie: SESSd14344a17ca11d13bda8baf612c0efa5=OBoeXKMQx3zmaZrS_v3FOP7_Ze66fYA61TGhtm9s0Qk
Content-Length: 34

{"title": "Updated dataset title"}

Response

{
    "nid": "75",
    "uri": "http://docker:32774/api/dataset/node/75"
}

Create resource

Request

POST http://docker:32774/api/dataset/node
Content-Type: application/json
Accept: application/json
X-CSRF-Token: XBWI44XD33XBIANLpyK-rtvRa0N5OcaC03qLx0VQsP4
Cookie: SESSd14344a17ca11d13bda8baf612c0efa5=OBoeXKMQx3zmaZrS_v3FOP7_Ze66fYA61TGhtm9s0Qk
Content-Length: 97

{
  "type": "resource",
  "field_dataset_ref": {"und": [{"target_id": "75"}]},
  "title": "Test Resource"
}

Response

{
    "nid": "76",
    "uri": "http://docker:32774/api/dataset/node/76"
}

Python client

The pydkan Python client interface with DKAN Dataset REST API DKAN Dataset web service and can be used to build on for your own applications. For more information please visit DKAN API Guide.

SDMX Overview

SDMX, which stands for Statistical Data and Metadata Exchange is an international initiative that aims at standardising and modernising (“industrialising”) the mechanisms and processes for the exchange of statistical data and metadata among international organisations and their member countries.

SDMX is sponsored by seven institutions including the Bank for International Settlements (BIS), the European Central Bank (ECB), Eurostat (Statistical Office of the European Union), the International Monetary Fund (IMF), the Organisation for Economic Cooperation and Development (OECD), the United Nations Statistical Division (UNSD), and the World Bank. Their purpose is to provide standards for the exchange of statistical information.

This standard describes and universalizes the way to exchange statistical data, and provides standard formats for data and metadata, content guidelines as well as IT architecture for exchange of data and metadata.

The most common terms used in SDMX and their definitions can be found below:

  • Dataset: a collection of related observations, organized according to a predefined structure.
  • Data Structure Definition (DSD): metadata describing the structure and organization of a dataset, the statistical concepts and attached to them code lists used within the dataset.
  • Dimensions: concepts that determine the dataset's "physical" structure.
  • Code lists: A code list is a predefined list from which some statistical coded concepts take their values. Each code list has the following properties: a) identifier (it provides a unique identification within the set of code lists specified by a structural definitions maintenance agency); b) name (also unique); c) description (a description of the purpose of the code list); and d) code value length (either an exact or a maximum number of characters and a type, i.e. numeric or alphanumeric).
  • Attributes: give additional information about the concepts used and do not affect the dataset structure itself.
  • Dataflow: A structure which describes categorizes and constrains the allowable content of a dataset that providers will supply for different reference periods.
  • Concept Schemes: The descriptive information for an arrangement or division of concepts into groups based on characteristics, which the objects have in common. A concept scheme is a maintained list of concepts that are used in key family and metadata structure definitions.

Definition from The official site for the SDMX community and Eurostat. For more information pleas visit The official site for the SDMX community.

Fusion Registry API / SDMX Web Servies

The Fusion Registry hosts a number of web services, including SDMX web services. SDMX Web Services include web services to retrieve and submit structures, data, and reference metadata. These web services comply with the SDMX Specification, supporting RESTful GET, and HTTP POST. The SDMX web services API supports multiple response formats for queries. The web services with no security imposed on them are accessed via ws/public/sdmxapi.

Ech web service is definied by following:

  • Web service entry point – the URL defining the entry point to the web service.
  • Resource – the resource that is being requested.
  • Path Parameters –additional URL path to define the required result.
  • Request Parameters – optional parameters used to further define the result (such as level of detail, format, further restrictions).
  • HTTP Header Parameters – Not part of the URL, but passed in on the http request. Used to pass further information to the server such as response language or response format.

Resources

In URL entrypoint the resource is used to determine which structure type is being queried. The most common resources:

  • dataflow - Returns all dataflows that match the subsequent path parameters.
  • data - Returns a dataset containing data that matches the subsequent path parameters.
  • agencyscheme - Returns all agency schemes that match the subsequent path parameters.
  • categoryscheme - Returns all category schemes that match the subsequent path parameters.
  • codelist - Returns all codelists that match the subsequent path parameters.
  • categorisation - Returns all categorisations that match the subsequent path parameters.
  • dataproviderscheme - Returns all data provider schemes that match the subsequent path parameters.
  • datastructure- Returns all data structure definitions that match the subsequent path parameters.

Path parameters

  • agencyID - The agency which owns the structure(s) all – default
  • structureID - The id of the structure(s) to be returned all – default
  • version - The version of the structure(s) to be returned latest – default
  • itemID - If the resource is to an item scheme, the item inside the scheme can be identified by this parameter

Query Data via REST API - Dataflow

  • Web service entrypoint for dataflow: /ws/public/sdmxapi/rest/dataflow
  • Query for get all dataflows: /ws/public/sdmxapi/rest/dataflow/all/all/latest/

Query Data via REST API - Dataflow metadata

  • Web service entrypoint for specific dataflow metadata: /ws/public/sdmxapi/rest/data
  • The string identifying the dataflow is dataflowRef. This parameter path is mandatory.

Syntax

  • The syntax is agency id, artefact id, version separated by a ",". For example: AGENCY_ID,FLOW_ID,VERSION
  • In case the string only contains one out of these 3 elements, it is considered to be the flow id, i.e.: all,FLOW_ID,latest
  • In case the string only contains two out of these 3 elements, they are considered to be the agency id and the flow id, i.e.: AGENCY_ID,FLOW_ID,latest

For more information please visit Fusion Registry Web Services API documentation.

0