Technical resources
Introductory Videos
- Introductory AusPIX DGGS Video
- Introduction Location Index Video
- Linked Data and DGGS Video
- CSIRO video demonstrating Loc-I Linked Data (2020, most of these capabilities are in GA now - September 2022)
AusPIX Spatially Linked-data APIs
AusPIX– providing metrics and spatial statistics (using a pre-calculated relationship grid built on DGGS technology)
AusPIX fastAPI tool (swagger page with many tools)
This API provides cross-referencing and statistical tools across geographical features. Exposes relationships between datasets to provide quick metrics and spatial statistics. Both human and machine to machine (automated) queries are easily processed, with results in json format. AusPIX handles points, lines, polygons and rasters.
AusPIX DGGS typical use cases:
Data drill: Discover the features recorded at a latitude/ longitude point. Useful for any query to discover the information at a given location. Very quick. Can answer such questions as:
- What is the name of the Electorate/ SA2/ LGA / Suburb / postcode / National Park that the location within?
- What is the risk related to earthquakes or cyclones at that location?
- Is the location likely to be ever flooded? (using WofS data)
- I have a list for Greater Glider possum sightings with latitude and longitude information. What is the approximate elevation above sea-level for these possum sightings?
- I need situational awareness for school locations and addresses. This tool allows me to discover which Electorate/ SA2/ LGA / Suburb / postcode etc a school or street address is located in.
- Supply this sort of information to higher level dashboards and APIs (via machine readable API calls using simple latitude and longitude parameters)
DGGS Cell query: discover the features and data available for a DGGS cell. Similar to the Data drill tool, returning information for a query on a DGGS AusPIX level 10 cell.
Apportionment Query: Discover the spatial relationship between a single feature in one geography and the features of any other geography. Quick analysis. User chooses the cross-analysis information they are looking for from countless options. Response is as a json string and includes a range of statistical results. Typical use cases include.
- I have an LGA. Tell me which SA1, SA2, SA3 etc, Postcodes, Suburbs, National Parks, Water Bodies, Streams, roads, Schools, Place names, Emergency Facilities etc etc are within or partly within (with % for apportionment).
- I am looking to buy a house and want to know the suburbs and the schools in them so I can choose. The user or a connected API can query suburbs for a list of schools in each one.
- I have a flood polygon that has been DGGS enabled and want some statistics on what land-uses have been affected within the flooded area. The user uses AusPIX to cross-reference the flooded area with land-use data that is already in the the AusPIX system.
- A bushfire has swept across large areas. The minister wants to have a list of the suburbs/postcodes/electorates that have been burnt along with the % figures to support a recovery initiatives.
User polygon: I have an area of interest as a WKT (well-known text) polygon. I need AusPIX cross-referenced information to describe and inform me about what features exist within that polygon. The API provides apportionment information for this WKT user-defined polygon. For example, the WKT polygon may represent the landfall of a cyclone and the Emergency Service wants information about what is threatened in the impact zone. The AusPIX system can quickly cross-reference the WKT polygon with any geographic data available in the AusPIX system.
Other uses: There are literally hundreds of use-cases involving cross-referencing. AusPIX can support crosswalks between any data that has been DGGS enabled. No need to wait for reprocessing, the AusPIX system based on DGGS cells includes all combinations, being able to relate to any combination.
Other AusPIX DGGS based APIs
- DGGS enablement API (AusPIX DGGS API tool to DGGS enable small-medium datasets (human and machine readable)
- Interactive DGGS Enablement Map Demonstrator to build your own AusPIX DGGS spatial reference for points, lines or polygons.
- Linked data API for each AusPIX DGGS cell (Provides human-readable and machine-readable access to cell information including Landing Page with map showing cell location and data describing location of cell corners and cell centroid. Parent cells and child cells are described.)
Semantically Linked data APIs
Providing for semantic linked-data queries, metadata information, provenance information, and individual feature mapping. These APIs source their information from RDF triple stores containing a copy of the source data. Features of these APIs are:
- standardised interoperability
- human readable landing page for each geographic feature,
- machine readability for automation,
- variety of alternate profiles for viewing and accessing data
- metadata, conformance, and vocab information
- searchable
- built for semantic queries
- cross dataset queries need to be done within the same triple store
Semantically linked data cannot natively provide apportionment figures. To work around this semantic systems sometime use different types of spatially linked databases (like PostGIS) and do spatial calculations to provide these apportionment figures (% relationships between features in different datasets). GeoSPARQL can also provide some of these calculations but the time takes is similar to a GIS package calculation doing the same thing. The AusPIX spatially linked APIs fill this gap to provide greater speed and flexibility.
Because these systems are built as machine-readable APIs, the network of information is available to API calls to gather exactly the information that is required.
Use cases for semantically linked-data using APIs
- A user has identified a feature (e.g an LGA) and needs to see it on a map, discover its attributes, and determine the age & provenance of the information. The search feature in the API allows this user to find the LGA and provide the information required. Note: the user has feature level access and doesn’t need to download and manipulate the whole dataset.
- A user quickly wants the geometry for Commonwealth Electorate. The user finds the Electorate in the Linked Data API, and copies the geometry (as geojson) to a map.
- A data scientist wants to semantically query Wikidata information against some GA Triple Store of RDF data. The scientist imports the required Wikidata triples and the GA/ABS RDF triples into a Wikibase instance and proceeds to discover trends and relationships for a study on Australian population dynamics. Since Wikidata and GA linked data are stored as RDF triples the schemas are identical – making the job a lot easier.
- To make data more usable under a Government directive regarding F.A.I.R principles, Geoscience Australia publishes its data as web-connected RDF triples. This allows the GA data to be part of systems that connect and query data from different sources. The RDF triples provide a common schema for rapid and standardised transfer of information. APIs written over the RDF triple stores allow diverse range of dashboards and informative web pages to be developed for special use cases. GA provides the data as Triples, builds some useful APIs inside GA, and allows others to build and maintain their own APIs over the GA world standard RDF data sources.
Available Semantic Linked-Data APIs in GA
- FSDF dataset suite - Electrical Infrastructure, Emergency Facilities, Floods, Placenames
- Vocabularies for FSDF and other datasets
- Australian Statistical Geography Standard (ASGS) in conjunction with ABS
- Australian Hydrological Geospatial Fabric Geofabric catchments, divisions and river regions
- Geocoded National Address File (GNAF)
- Linked Data Cashe API for loading datasets to be cross-referenced, data will need to be added and removed as required. This is where data is loaded for temporary cross-referencing work. (NB as the technology stands today, for cross-referencing to work all relevant RDF triples must be located in the same RDF Triple Store).
Loci API products adapted for and hosted in Geoscience Australia
- Loci-I style Data “Explorer” built over Geoscience Australia Triple stores - an interactive feature discovery tool – (not working within GA, but working fine externally)
- Explorer GitHub code – code to build “Explorer”
- Loci style “Iderdown” - use this code to run locally in Jupyter Notebook – a tool to find ID’s of geographical features using RDF triple stores and data source.
- Placenames LD API – Composite Gazetteer of Australia (older version)
- SPARQL endpoint API has a SPARQL endpoint so that SPARQL can be used to query the database. Sometime CQL or other query methods are provided as alternative query languages. Examples are provided including examples of how machine-readable queries are built.
Technical Documentation
- Spatial enablement for data and metadata User Guides and Best Practices
- AusPIX Conceptual Framework for Spatial Feature Cross-referencing based on DGGS Location
- OGC API - Features - Part 1: Core
- Location Index - Super Model specification by Surround Australia Pty Ltd for Geoscience Australia
- Jupyter Notebooks (CSIRO) Loci notebooks (previous development – AusPIX DGGS integration with Semantic Linked data included)
- Location Index Principles, Technical Design, Deployment Details (older development information)
- Location Index Social Architecture Report (Loc-I initial project stakeholder engagement)
Training Plans
- Location Index Training Plan for Surround build -Surround Digital Atlas Linked Data training plan for loading and modifying datasets.
- Linked data Training Plan source on GitHub
- Cookbook on how to build a Linked Data API and publish it on AWS
- AusPIX Conceptual Framework for Data Integration based on DGGS Location
Ontologies
Concepts and categories with their properties and the relationships between them - providing data models and vocabularies.
- AusPIX Ontology
- Place Names Ontology
- GeoSPARQL Extensions Ontology
- Location Index Ontology
- ASGS Ontology - ABS Structures and non-ABS structures
- Geofabric Ontology
- GNAF Ontology
Vocabulary Documentation and Tools
- FSDF VocPrez
- VocPrez overview vocabulary reader tool - human and machine readable
- FSDF SPARQL endpoint - interact with SPARQL queries
- VocPrez information video 2019
- GA themed Prez Adds the ASGS, GA, and GNAF themes to a Prez docker image. NOTE: Bitbucket resources require access authorisation
Provenance development
- Provenance demonstration on Jupyter Notebooks
- Sankey Diagram Provenance Tool – visualisation of province information
- GitHub code for Sankey Provenance Tool
- Bitbucket code for Sankey Diagram
- The PROV Namespace W3C standards and Data Models
Semantic Validation tools
- Validator Tool – Dashboard user interface using pySHACL - to suit many styles of RDF.
Code Repositories GitHub and Bitbucket
NOTE: Bitbucket resources require access authorisation.
- Code for New Surround Australia Pty Ltd Linked Data API
- Code repository for RDF validation tools online
- Code Repository for Linked-Data workflows (ETL) to build RDF triples from source data
- Code Repository for Location Index Ontology
- Code Repository for AusPIX DGGS Ontology
- Third party Python code that calculates DGGS cells for AusPIX (on PyPi)
- Developmental Scripts for AusPIX DGGS functions
- AusPIX PostgreSQL Crosswalk Tables source code - Code to build and query AusPIX DGGS Crosswalk Tables.
- BitBucket code repositories for AusPIX DGGS
- The GitHub code repository for the Interactive DGGS Enablement Map
- Placenames API source code (old version) (The GitHub repository for the code for Placenames Linked Data API)
- Code for Lambda Serverless API’s for AusPIX running on AWS (Bitbucket)
- Code to Build and visualise AusPIX processing Tiles for Australia (built on DGGS level 4 cells, approximately 140km by 140km tiles to suit multi-CPU processing)
- RDF Linked Datasets available through their Linked-Data APIs (with landing page for each feature, human readable and machine readable)
- Code Repositiory AusPIX engine (our original copy in Geoscience Australia infrastructure)
- Tableau Dashboards (Built from AusPIX crosswalk tables. Tableau Software can access the AWS PostgreSQL AusPIX crosswalk data to display information in dashboards) Tableau cannot be automated to the same extent as our AusPIX or Semantic workflows.