The share is the catalogue
There is no manifest file to maintain. build_catalogue()
walks \\ABMI-DATA2\science\spatial_data, parses the readme
stored beside each dataset, and assembles a table from what it finds.
Documenting a dataset means editing its readme; nothing has to be
registered anywhere, and the catalogue cannot drift from the data.
Folders follow the ISO 19115 topic categories and readmes follow the ABMI spatial metadata template, both documented in the geospatial catalog and management guide.
A folder is catalogued as a layer when it holds a
readme written to the product or variant template. Theme folders carry a
shorter
Category/Description/Examples
readme and are reported by list_themes() instead. Products
sit directly under their topic category, so each layer has both a short
name (fab_dem) and a full id
(elevation/fab_dem). A product may split its record across
a product readme and one readme per variant subfolder, in which case the
variant is the layer and its id carries the variant folder
too
(geoscientificInformation/soilgrids_250_v2_ab/abmi1km).
Two top-level folders no scan looks at are _temp, the
share’s scratch area, and _deprecated, where superseded
products are parked. Work in progress, staging copies, and exports live
in the first; products that have been replaced live in the second. The
contents of neither are catalogued as layers or counted against the
share by check_metadata() — a layer only becomes
discoverable once it is filed under a theme, and it stops being
discoverable once it is moved to _deprecated.
Pointing at the share
spatial_root() reports where the catalogue reads from
and errors if the share is unreachable.
spatial_root()
#> [1] "//ABMI-DATA2/science/spatial_data"To work from a mirror, a mapped drive, or a local copy, set an option or an environment variable:
options(sciSpatialR.spatial_root = "D:/spatial_data")
Sys.setenv(SCISPATIALR_SPATIAL_ROOT = "D:/spatial_data")Browsing
Start with the themes, which show the topic categories and how much sits under each:
14 themes, 20 layers catalogued
theme description n_layers
elevation Height above or below sea level. 5
inlandWaters Inland water features, drainage systems, and thei… 4
imageryBaseMapsEarthCover Base maps. 3
biota Flora and/or fauna in natural environments. 2
geoscientificInformation Information pertaining to earth sciences. 2
location Positional information and services. 2
boundaries Legal land descriptions. 1
transportation Means and aids for conveying persons and/or goods. 1
.git <NA> 0
climatologyMeteorologyAtmosphere Processes and phenomena of the atmosphere. 0
economy Economic activities, conditions, and employment. 0
environment Environmental resources, protection, and conserva… 0
farming Rearing of animals and/or cultivation of plants. 0
figures <NA> 0 Every ISO 19115 topic category appears, including the ones nothing is
filed under yet, so the empty folders read as available rather than
missing. A theme folder with no readme of its own would still appear,
with no description, so an undocumented theme stays visible rather than
dropping out of the listing. _temp and
_deprecated are the exceptions, and are absent: they are
scratch and retirement space rather than topic categories.
The returned table carries an examples column as well,
too long to tabulate but useful when deciding where a new dataset
belongs.
Then list the layers:
20 layers in 8 themes
id title year resolution_m data_type
biota/grassland_inventory Grassland Inventory for Alberta, Manitoba, a… 2024 30.00 raster
biota/natural_regions_subregions_of_alberta Natural Regions and Subregions of Alberta 2022 NA vector
boundaries/alberta AB2020_provincial_boundary 2023 NA vector
elevation/fab_dem FABDEM – Forest And Buildings Removed Copern… 2018 100.00 raster
elevation/geomorpho90 Alberta Geomorphometric Layers (Geomorpho90m) 2023 90.00 raster
elevation/nrcan_mrdem_dsm Medium Resolution Digital Elevation Model (M… 2006 30.00 raster
elevation/nrcan_mrdem_dtm Medium Resolution Digital Elevation Model (M… 2006 30.00 raster
elevation/nrcan_mrdem_dtm_hillshade Medium Resolution Digital Elevation Model (M… 2006 30.00 raster
geoscientificInformation/hihydrosoil_v2_ab/abmi1km HiHydroSoil v2.0 2020 1000.00 raster
geoscientificInformation/soilgrids_250_v2_ab/abmi1km SoilGrids 2.0 global soil property predictio… 2020 1000.00 raster
imageryBaseMapsEarthCover/landsat_summer_mean_indices_2000_2024 Landsat Time Series - Alberta Mean Spectral … 2024 30.00 raster
imageryBaseMapsEarthCover/modis_land_cover_dynamics_2001_2023 MODIS Annual Land Cover Dynamics (MCD12Q2) -… 2023 500.00 raster
imageryBaseMapsEarthCover/scanfi_v1.2 SCANFI: Spatialized Canadian National Forest… 2020 30.00 raster
inlandWaters/archydro2 Alberta ArcHydro Phase 2 Data 1996 100.00 vector
inlandWaters/dynamicSurfaceWaterMaps Dynamic Surface Water Maps of Canada from 19… 2023 NA raster
inlandWaters/hydrologically adjusted elevations Height Above Nearest Drainage (HAND) - Hydro… 2024 92.77 raster
inlandWaters/topographic_wetness_index Topographic Wetness Index (TWI) 2024 92.77 raster
location/GRID1SQKM_AB2020_gdb GRID1SQKM_AB2020 NA 1000.00 vector
location/GRID1SQKM_AB2020_raster GRID1SQKM_AB2020_raster 2026 1000.00 raster
transportation/government_of_alberta_access_layers Access and Facility Roads - Alberta 2023 NA vector Nothing under _temp or _deprecated is
listed, documented or not — the Sentinel-2 composites staged in the
first are invisible to the catalogue until they are filed under a theme,
and a product moved to the second disappears from it.
Or narrow the listing to one theme:
list_layers(theme = "elevation")5 layers in 1 theme
id title year resolution_m data_type
elevation/fab_dem FABDEM – Forest And Buildings Removed Copern… 2018 100 raster
elevation/geomorpho90 Alberta Geomorphometric Layers (Geomorpho90m) 2023 90 raster
elevation/nrcan_mrdem_dsm Medium Resolution Digital Elevation Model (M… 2006 30 raster
elevation/nrcan_mrdem_dtm Medium Resolution Digital Elevation Model (M… 2006 30 raster
elevation/nrcan_mrdem_dtm_hillshade Medium Resolution Digital Elevation Model (M… 2006 30 raster list_layers() prints a summary and returns the manifest
invisibly. Assign it, or pass verbose = FALSE, to work with
the full table — one row per layer, carrying the parsed template fields
alongside n_files, size_mb,
data_type, path, and readme.
cat_df <- list_layers(verbose = FALSE)
dim(cat_df)
#> [1] 18 40Forty columns is too wide to print whole; a few of them give the shape of the table:
cat_df[1:5, c("name", "theme", "year", "resolution_m",
"n_files", "size_mb", "data_type")] name theme year resolution_m n_files size_mb data_type
1 natural_regions_subregions_of_alberta biota 2022 NA 1 11.2 vector
2 grassland_inventory biota 2024 30 3 30110.4 raster
3 alberta boundaries 2023 NA 1 0.4 vector
4 fab_dem elevation 2018 100 1 11272.9 raster
5 geomorpho90 elevation 2023 90 1 11197.9 rasterThe full set of fields — the parsed template alongside the columns the file scan adds:
names(cat_df) [1] "id" "name" "theme" "product_id"
[5] "variant" "title" "topic_category" "keywords"
[9] "year" "resolution" "resolution_m" "xmin"
[13] "xmax" "ymin" "ymax" "crs"
[17] "crs_name" "datum" "vertical_crs" "publication_date"
[21] "start_date" "end_date" "format" "size"
[25] "use_constraints" "access_constraints" "contact_name" "contact_email"
[29] "doi" "online_resource" "metadata_date" "abstract"
[33] "purpose" "credits" "lineage" "citation"
[37] "n_files" "size_mb" "data_type" "path"
[41] "readme" "product_readme"product_id and variant are what a split
record adds. A product documented in one readme has variant
NA and a product_id equal to its
id. A product whose data sits in variant subfolders
contributes one row per variant instead of a row of its own:
id names the variant folder, product_id the
product above it, and readme and
product_readme point at the two halves of the record.
id product_id variant resolution_m crs
geoscientificInformation/hihydrosoil_v2_ab/abmi1km …/hihydrosoil_v2_ab abmi1km 1000 EPSG:3400
geoscientificInformation/soilgrids_250_v2_ab/abmi1km …/soilgrids_250_v2_ab abmi1km 1000 EPSG:3400The resolution and CRS above come from the variant records; the
titles and licences those rows also carry come from the product records.
layer_meta() returns the two read together, so nothing is
lost by the split.
The catalogue as it stands
The table below is a frozen scan of the share, rendered when this
vignette was built. It is what list_layers(verbose = FALSE)
returns, narrowed to the columns worth tabulating — run the call
yourself for the live version and the other thirty-odd columns.
| theme | name | title | year | res (m) | type | size (MB) |
|---|---|---|---|---|---|---|
| biota | grassland_inventory | Grassland Inventory for Alberta, Manitoba, and Saskatchewan (2023) | 2024 | 30.00 | raster | 30,110.4 |
| biota | natural_regions_subregions_of_alberta | Natural Regions and Subregions of Alberta | 2022 | vector | 11.2 | |
| boundaries | alberta | AB2020_provincial_boundary | 2023 | vector | 0.4 | |
| elevation | fab_dem | FABDEM – Forest And Buildings Removed Copernicus Global DEM (30 m) | 2018 | 100.00 | raster | 11,272.9 |
| elevation | geomorpho90 | Alberta Geomorphometric Layers (Geomorpho90m) | 2023 | 90.00 | raster | 11,197.9 |
| elevation | nrcan_mrdem_dsm | Medium Resolution Digital Elevation Model (MRDEM) - DSM Cloud Optimized GeoTIFF (COG) | 2006 | 30.00 | raster | 56,895.8 |
| elevation | nrcan_mrdem_dtm | Medium Resolution Digital Elevation Model (MRDEM) - DTM Cloud Optimized GeoTIFF (COG) | 2006 | 30.00 | raster | 57,503.7 |
| elevation | nrcan_mrdem_dtm_hillshade | Medium Resolution Digital Elevation Model (MRDEM) - DTM Hillshade Cloud Optimized GeoTIFF (COG) | 2006 | 30.00 | raster | 13,648.1 |
| geoscientificInformation | hihydrosoil_v2_ab | HiHydroSoil v2.0 | 2020 | 1,000.00 | raster | 220.5 |
| geoscientificInformation | soilgrids_250_v2_ab | SoilGrids 2.0 global soil property predictions | 2020 | 1,000.00 | raster | 174.0 |
| imageryBaseMapsEarthCover | landsat_summer_mean_indices_2000_2024 | Landsat Time Series - Alberta Mean Spectral Indices (2000-2024) | 2024 | 30.00 | raster | 2,772,122.1 |
| imageryBaseMapsEarthCover | modis_land_cover_dynamics_2001_2023 | MODIS Annual Land Cover Dynamics (MCD12Q2) - 500m Phenology | 2023 | 500.00 | raster | 2,529.0 |
| imageryBaseMapsEarthCover | scanfi_v1.2 | SCANFI: Spatialized Canadian National Forest Inventory | 2020 | 30.00 | raster | 34,349.8 |
| inlandWaters | archydro2 | Alberta ArcHydro Phase 2 Data | 1996 | 100.00 | vector | 680.3 |
| inlandWaters | dynamicSurfaceWaterMaps | Dynamic Surface Water Maps of Canada from 1984-2023 Landsat Satellite Imagery | 2023 | raster | 21,796.5 | |
| inlandWaters | hydrologically adjusted elevations | Height Above Nearest Drainage (HAND) - Hydrologically Adjusted Elevations | 2024 | 92.77 | raster | 239.0 |
| inlandWaters | topographic_wetness_index | Topographic Wetness Index (TWI) | 2024 | 92.77 | raster | 1,193.0 |
| location | GRID1SQKM_AB2020_gdb | GRID1SQKM_AB2020 | 1,000.00 | vector | 322.8 | |
| location | GRID1SQKM_AB2020_raster | GRID1SQKM_AB2020_raster | 2026 | 1,000.00 | raster | 0.0 |
| transportation | government_of_alberta_access_layers | Access and Facility Roads - Alberta | 2023 | vector | 1,399.7 |
Blank cells are fields the readme leaves unfilled, which is what
keeps a layer out of the matching find_layer() filter.
What variables a product holds
list_layers() says which layers exist;
list_variables() says what is inside them. It
reads the band blocks out of each product readme, so the variables
available for extraction can be browsed without opening a readme or a
raster.
list_variables(product = "soilgrids_250_v2_ab")61 variables across 1 product
product band measure units
soilgrids_250_v2_ab bdod_0-5cm_mean Bulk density of the fine earth fraction kg/dm3
soilgrids_250_v2_ab bdod_5-15cm_mean Bulk density of the fine earth fraction kg/dm3
soilgrids_250_v2_ab bdod_15-30cm_mean Bulk density of the fine earth fraction kg/dm3
soilgrids_250_v2_ab bdod_30-60cm_mean Bulk density of the fine earth fraction kg/dm3
soilgrids_250_v2_ab bdod_60-100cm_mean Bulk density of the fine earth fraction kg/dm3
soilgrids_250_v2_ab bdod_100-200cm_mean Bulk density of the fine earth fraction kg/dm3
soilgrids_250_v2_ab cec_0-5cm_mean Cation exchange capacity of the soil cmol(c)/kg
soilgrids_250_v2_ab cec_5-15cm_mean Cation exchange capacity of the soil cmol(c)/kg
...Bands belong to the product rather than to a variant — resampling a layer does not change what its bands measure — so a product with several variants reports one set of rows, not one set per variant.
The compact view above shows two of the columns; the returned table
also carries scale, valid_range, and
description. Called with no arguments it covers the whole
catalogue, which makes it searchable — every band measuring pH, wherever
it lives:
v <- list_variables(verbose = FALSE)
v[grepl("pH", v$measure, fixed = TRUE),
c("name", "band", "units")] name band units
71 soilgrids_250_v2_ab phh2o_0-5cm_mean pH
72 soilgrids_250_v2_ab phh2o_5-15cm_mean pH
73 soilgrids_250_v2_ab phh2o_15-30cm_mean pH
74 soilgrids_250_v2_ab phh2o_30-60cm_mean pH
75 soilgrids_250_v2_ab phh2o_60-100cm_mean pH
76 soilgrids_250_v2_ab phh2o_100-200cm_mean pHtheme narrows the listing the same way it does
elsewhere. A product whose readme documents no bands is listed with
band NA rather than dropped, so the gap stays
visible the way check_metadata() keeps missing template
fields visible.
Finding a layer
find_layer() searches by what a layer is rather
than where it lives. keyword matches the folder name,
title, keywords, topic category, and abstract:
find_layer(keyword = "elevation")10 layers in 4 themes
id title year resolution_m data_type
biota/natural_regions_subregions_of_alberta Natural Regions and Subregions of Alberta 2022 NA vector
elevation/fab_dem FABDEM – Forest And Buildings Removed Copern… 2018 100.00 raster
elevation/geomorpho90 Alberta Geomorphometric Layers (Geomorpho90m) 2023 90.00 raster
elevation/nrcan_mrdem_dsm Medium Resolution Digital Elevation Model (M… 2006 30.00 raster
elevation/nrcan_mrdem_dtm Medium Resolution Digital Elevation Model (M… 2006 30.00 raster
elevation/nrcan_mrdem_dtm_hillshade Medium Resolution Digital Elevation Model (M… 2006 30.00 raster
imageryBaseMapsEarthCover/modis_land_cover_dynamics_2001_2023 MODIS Annual Land Cover Dynamics (MCD12Q2) -… 2023 500.00 raster
inlandWaters/archydro2 Alberta ArcHydro Phase 2 Data 1996 100.00 vector
inlandWaters/hydrologically adjusted elevations Height Above Nearest Drainage (HAND) - Hydro… 2024 92.77 raster
inlandWaters/topographic_wetness_index Topographic Wetness Index (TWI) 2024 92.77 raster Note the first hit: a layer filed under biota matched on
its abstract, which is the point of searching metadata rather than
folder names.
Filters combine, and each takes either a single value or a
c(min, max) range:
find_layer(year = c(2020, 2024))
find_layer(resolution = c(0, 120))
find_layer(extent = c(-120, -110, 49, 60)) # decimal degrees
find_layer(theme = "elevation", resolution = 30)A filter narrows to layers known to match. Resolution, year,
and extent come from the readme, so a layer whose readme leaves the
field blank is excluded rather than guessed at —
check_metadata() will show you which readmes are keeping a
layer out.
Loading a layer
get_layer() reads the data with terra, returning a
SpatRaster or a SpatVector depending on the
format:
twi <- get_layer("topographic_wetness_index")
nsr <- get_layer("natural_regions_subregions_of_alberta")Pass return_path = TRUE when you want the file rather
than the object — to hand it to another tool, or to open it with your
own arguments:
get_layer("fab_dem", return_path = TRUE)
#> [1] "//ABMI-DATA2/science/spatial_data/elevation/fab_dem/fab_dem_us_canada.tif"Some folders hold several datasets. layer_files() shows
what is there, and file picks one:
layer_files("grassland_inventory")
#> [1] ".../alberta_grassland_classification_2023.tif"
#> [2] ".../manitoba_grassland_classification_2023.tif"
#> [3] ".../saskatchewan_grassland_classification_2023.tif.tif"
get_layer("grassland_inventory", file = "alberta")Asking for a multi-file layer without file is an error
that lists the candidates, so you never silently get the wrong raster.
Shapefile sidecars (.dbf, .shx,
.prj) are hidden; pass all = TRUE to see every
file in the folder.
Short names work when unambiguous. If two themes hold a folder of the same name, use the full id:
get_layer("elevation/fab_dem")Taking a local copy
get_layer() reads a layer where it sits, over the
network. To work offline or to avoid server latency,
download_layer() copies one to a folder on your local
machine:
download_layer("fab_dem", "D:/local_spatial")
#> Copying elevation/fab_dem: 3 files, 11.0 GB to
#> D:/local_spatial/fab_dem ...
#> Copied 3 files (11.0 GB) in 9m 12s; 0 already up to date.
#> Verified 3 files against the source on size and md5 checksum.
#> Copy log written to D:/local_spatial/fab_dem/_download_log.txtEverything in the layer folder comes down, not just the data: the
readme that documents it, and any ancillary files. The shapefile
sidecars layer_files() hides come too, so the copy
opens.
The arguments:
| Argument | Does what |
|---|---|
name |
Layer name or catalogue id, as for get_layer()
|
dest |
Local folder to copy into; created if it does not exist |
dry_run |
Report what would be copied, and what it comes to, writing nothing |
verify |
Read each copy back and check size and md5 against the source.
TRUE by default |
overwrite |
Copy every file again rather than skipping unchanged ones |
keep_theme |
Reproduce the share’s full path, ISO topic category included |
quiet |
Suppress the progress messages |
Test before copying
A copy from the share can be large, so start with
dry_run = TRUE. It does the whole search and sizing and
prints summary to console:
download_layer("scanfi_v1.2", "D:/local_spatial", dry_run = TRUE)
#> Dry run — nothing was written.
#> 14 files (22.6 GB) would be copied to D:/local_spatial; 0 already
#> up to date.
#> file size part status
#> scanfi_v1.2/_readme.txt 4.1 KB layer would copy
#> scanfi_v1.2/scanfi_index.tif 1.8 GB layer would copy
#> ...The same table comes back as a data.frame, so you can
total it or filter it yourself before committing to the copy.
Where the files land
The copy is rooted at the product folder, which
lands directly in dest. The ISO topic category is the
share’s filing system rather than part of the dataset, so it is left
behind — elevation/fab_dem on the share arrives as
D:/local_spatial/fab_dem.
Everything from the product folder down is kept, so where a product splits its record the product readme still lands one level up from the variant, exactly as the two sit on the share:
p <- download_layer(
"geoscientificInformation/soilgrids_250_v2_ab/abmi1km",
"D:/local_spatial"
)
p[, c("part", "destination", "status")]
#> part destination status
#> layer D:/local_spatial/soilgrids_250_v2_ab/abmi1km/readme.txt copied
#> layer D:/local_spatial/soilgrids_250_v2_ab/abmi1km/soil.tif copied
#> product D:/local_spatial/soilgrids_250_v2_ab/readme.txt copiedThe variant readme carries the resolution, CRS, and derivation, while
the product readme carries the title, licence, and citation. A variant
copied on its own would arrive with half its provenance. Sibling
variants are separate layers and are not swept up; ask for one
by its own id, since the name they share is
the product’s.
geomorpho90 publishes two: the provider’s native 90 m
grid, and a copy resampled to the ABMI 1 km grid. Downloading both puts
them side by side under the one product folder:
download_layer("elevation/geomorpho90/abmi1km", "D:/local_spatial")
#> Copying elevation/geomorpho90/abmi1km: 3 files, ... ...
#> Copied 3 files (...); 0 already up to date.
p <- download_layer("elevation/geomorpho90/native", "D:/local_spatial")
p[, c("part", "status")]
#> part status
#> layer copied
#> layer copied
#> product skippedThe product row comes back skipped on the
second download. Both variants share one product record, so the second
finds it already there rather than writing it twice, and the copy ends
up with the single shared readme the share has:
D:/local_spatial/geomorpho90/
|-- readme.txt <- the product record, shared
|-- abmi1km/
| |-- readme.txt <- this variant's own record
| |-- geomorpho90_abmi1km.tif
| `-- _download_log.txt
`-- native/
|-- readme.txt
|-- geomorpho90_native.tif
`-- _download_log.txtRe-running is cheap
A file already present and matching its source on size and modification time is skipped, so an interrupted copy resumes rather than restarts:
download_layer("fab_dem", "D:/local_spatial")
#> elevation/fab_dem is already up to date in D:/local_spatial (3 files).overwrite = TRUE forces a fresh copy regardless, which
is what to reach for when you suspect the local copy rather than the
transfer:
download_layer("fab_dem", "D:/local_spatial", overwrite = TRUE)
#> Copying elevation/fab_dem: 3 files, 11.0 GB to
#> D:/local_spatial/fab_dem ...
#> Copied 3 files (11.0 GB) in 9m 08s; 0 already up to date.
#> Verified 3 files against the source on size and md5 checksum.Keeping a local mirror
Everything above copied into D:/local_spatial in the
default layout (keep_theme = TRUE), which is fine for
copying a single dataset to work on. To produce a mirror (a
folder the catalogue itself can read) needs
keep_theme = TRUE, so the topic category folder is
kept.
download_layer("elevation/fab_dem", "D:/local_mirror",
keep_theme = TRUE)
download_layer(
"geoscientificInformation/soilgrids_250_v2_ab/abmi1km",
"D:/local_mirror",
keep_theme = TRUE
)Each lands at the path its id names, the product readme
one level above the variant as on the share:
D:/local_mirror/
|-- elevation/
| `-- fab_dem/
| |-- readme.txt
| |-- fab_dem_us_canada.tif
| `-- _download_log.txt
`-- geoscientificInformation/
`-- soilgrids_250_v2_ab/
|-- readme.txt <- the product record
`-- abmi1km/
|-- readme.txt <- this variant's own record
|-- soilgrids_abmi1km.tif
`-- _download_log.txtPoint spatial_root() at the mirror and the package works
offline against it. The split records survive, so a variant is still
catalogued as its own layer and still inherits the title and licence
from the product readme above it:
options(sciSpatialR.spatial_root = "D:/local_mirror")
list_layers()
#> 2 layers in 2 themes
#>
#> id title
#> elevation/fab_dem FABDEM – Forest…
#> geoscientificInformation/soilgrids_250_v2_ab/abmi1km SoilGrids 2.0 …
get_layer("soilgrids_250_v2_ab")Switch back to default directory for below code chunks
options(sciSpatialR.spatial_root = NULL)
spatial_root()
#> [1] "//ABMI-DATA2/science/spatial_data"Copy info saved beside the data
Every copy writes _download_log.txt into the layer’s
folder, next to the readme:
cat(readLines("D:/local_spatial/fab_dem/_download_log.txt"),
sep = "\n")
#> ================================================================
#> sciSpatialR download log
#> ================================================================
#>
#> Layer: elevation/fab_dem
#> Title: Forest and Buildings Removed DEM
#> Source: //ABMI-DATA2/science/spatial_data/elevation/fab_dem
#> Destination: D:/local_spatial/fab_dem
#> Downloaded: 2026-09-24 09:14:02
#> By: bcasey
#> Package: sciSpatialR 0.1.0
#> Elapsed: 9m 12s
#> Verified: size and md5 checksum against the source
#>
#> FILES
#> ----------------------------------------------------------------
#> status verified bytes md5 file
#> copied ok 11820476416 d41d8cd9... fab_dem/fab_dem.tif
#> ...
#>
#> SUMMARY
#> ----------------------------------------------------------------
#> 3 copied, 0 already up to date, 0 failed
#> 11.0 GB in 9m 12s (20.4 MB/s)
#> All 3 files match their source on size and md5 checksum.By default each copied file is read back and compared to its source
on size and md5 checksum before the log records it as sound, and the
same verdict comes back in the verified and
md5 columns of the manifest. Only files copied in that run
are checked. Verification re-reads each copied file from the share, so
it roughly doubles the time a large copy takes; pass
verify = FALSE to skip it.
Runs append to the log rather than replacing it, so a layer re-copied over several sessions keeps its whole history. A run that copies nothing leaves it alone, so the file records real copy events rather than filling up with no-ops.
Reading the metadata
layer_meta() prints a layer’s provenance, licence, and
contact details, and returns the parsed fields invisibly:
layer_meta("fab_dem")<sciSpatialR metadata>
file: //ABMI-DATA2/science/spatial_data/elevation/fab_dem/readme.txt
Title FABDEM – Forest And Buildings Removed Copernicus Global DEM…
Abstract FABDEM is a global, bare-earth digital elevation model deri…
Purpose To provide a globally consistent, near–bare-earth digital e…
Topic geoscientificInformation
Keywords digital elevation model, DEM, bare-earth, terrain, topograp…
Resolution 3.23 arc-seconds (~100 m at the equator)
Published 2022-01-01
Start 2010-01-01
End 2018-12-31
Lineage FABDEM was derived from the Copernicus GLO-30 Digital Eleva…
Format GeoTIFF
Size Global dataset; size varies by tile (1° × 1° tiles grouped …
Use Creative Commons Attribution–NonCommercial–ShareAlike 4.0 I…
Access None. Data is publicly available subject to license terms.
Contact Laurence Hawker
DOI 10.1088/1748-9326/ac4d4f
Source https://doi.org/10.1088/1748-9326/ac4d4f
Updated 2026-01-23Values are truncated for display only; the returned object holds them
in full. Fields the readme left blank are omitted from the printout —
here Credits and Email, which
check_metadata() reports below.
Every Label: value line in the readme is captured, not
just the template fields, so anything a readme records is reachable.
Labels become snake_case keys, and sub-fields indented
under a parent are prefixed with it:
md <- layer_meta("fab_dem", print = FALSE)
md$use_constraints
md$extent_west_bounding_coordinate
md$temporal_extent_start_dateread_metadata() does the same for a readme you name by
path, and as_metadata_row() flattens the result to a
one-row data.frame with the numeric fields coerced —
resolution to metres, bounding coordinates to decimal degrees, dates to
a year:
md <- read_metadata("path/to/readme.txt")
as_metadata_row(md)Unfilled template placeholders ([Data Title]) and
non-values ("Not Specified") are returned as
NA rather than as text, so a copied-but-unedited readme
reads as missing rather than documented.
Auditing the metadata
check_metadata() scores every layer against the required
template fields and lists the data folders that have no readme at
all:
28 layers audited, 8 with no readme
id n_missing complete missing
…here/climate_na/fab_dem_us_canada_int/Normal_1991_2020 NA 0.000 readme
…yAtmosphere/climate_na/fab_dem_us_canada_int/Year_2000 NA 0.000 readme
…yAtmosphere/climate_na/fab_dem_us_canada_int/Year_2005 NA 0.000 readme
…yAtmosphere/climate_na/fab_dem_us_canada_int/Year_2010 NA 0.000 readme
…yAtmosphere/climate_na/fab_dem_us_canada_int/Year_2015 NA 0.000 readme
…yAtmosphere/climate_na/fab_dem_us_canada_int/Year_2020 NA 0.000 readme
…yAtmosphere/climate_na/fab_dem_us_canada_int/Year_2024 NA 0.000 readme
inlandWaters/archydro2_merged NA 0.000 readme
biota/grassland_inventory 8 0.579 purpose, xmin, xmax, ymin, ymax, use_constra…
boundaries/alberta 7 0.632 abstract, credits, resolution, use_constrain…
location/GRID1SQKM_AB2020_gdb 7 0.632 abstract, credits, publication_date, use_con…
imageryBaseMapsEarthCover/scanfi_v1.2 6 0.684 purpose, xmin, xmax, ymin, ymax, lineage
biota/natural_regions_subregions_of_alberta 4 0.789 xmin, xmax, ymin, ymax
elevation/fab_dem 2 0.895 credits, contact_email
geoscientificInformation/soilgrids_250_v2_ab/abmi1km 2 0.895 access_constraints, contact_email
location/GRID1SQKM_AB2020_raster 2 0.895 credits, access_constraints
transportation/government_of_alberta_access_layers 2 0.895 resolution, contact_name
elevation/nrcan_mrdem_dsm 1 0.947 access_constraints
elevation/nrcan_mrdem_dtm 1 0.947 access_constraints
elevation/nrcan_mrdem_dtm_hillshade 1 0.947 access_constraints
geoscientificInformation/hihydrosoil_v2_ab/abmi1km 1 0.947 contact_email
inlandWaters/dynamicSurfaceWaterMaps 1 0.947 resolution
elevation/geomorpho90 0 1.000
…seMapsEarthCover/landsat_summer_mean_indices_2000_2024 0 1.000
…BaseMapsEarthCover/modis_land_cover_dynamics_2001_2023 0 1.000
inlandWaters/archydro2 0 1.000
inlandWaters/hydrologically adjusted elevations 0 1.000
inlandWaters/topographic_wetness_index 0 1.000 (Abridged; folders with no readme sort first, then the least complete
readmes.) Rows with missing = "readme" are folders holding
spatial data that the catalogue cannot see at all — they are the first
thing to fix. Deep ids are trimmed from the front when printed, since
the last segment is what distinguishes one from another; the returned
table holds them in full.
Nothing under _temp or _deprecated is
audited either, so staging exports and retired products are not counted
as undocumented data. The audit stays a list of real gaps.
Use detail = TRUE for one row per missing field, which
is easier to tabulate:
check_metadata(detail = TRUE)
check_metadata(theme = "elevation", detail = TRUE)5 missing fields across 4 layers
id theme field
elevation/fab_dem elevation credits
elevation/fab_dem elevation contact_email
elevation/nrcan_mrdem_dsm elevation access_constraints
elevation/nrcan_mrdem_dtm elevation access_constraints
elevation/nrcan_mrdem_dtm_hillshade elevation access_constraintsBoth forms print the compact view above. Pass
verbose = FALSE for the plain data.frame
instead — every column, and the missing list in full rather
than truncated to the width of the printout:
gaps <- check_metadata(verbose = FALSE)
gaps$missing[gaps$id == "biota/grassland_inventory"]
#> [1] "purpose, xmin, xmax, ymin, ymax, use_constraints, access_constraints, contact_email"Caching
Scanning a network share is the slow step, so the manifest is built once per session and reused. After editing a readme or adding data, rescan:
build_catalogue(refresh = TRUE)build_catalogue(files = FALSE) skips the file inventory
for a faster, metadata-only scan; n_files,
size_mb, and data_type are then
NA. Any argument accepted by build_catalogue()
can be passed through the query functions:
list_layers(root = "D:/spatial_data", refresh = TRUE)Adding a field to the template
Readmes are parsed generically rather than against a fixed schema, so
a field you add to the template is readable immediately, with no change
to the package. Every Label: value line is captured, and a
sub-field indented under a parent label is namespaced by it. Given this
block:
Coordinate Reference System:
Name: NAD83 / Alberta 10-TM (Forest)
Authority Code: EPSG:3400
Datum: North American Datum 1983
Projection: Transverse Mercator
Vertical CRS: CGVD2013every part is reachable at once:
md <- layer_meta("some_layer", print = FALSE)
md$coordinate_reference_system_authority_code
#> [1] "EPSG:3400"
md$coordinate_reference_system_projection
#> [1] "Transverse Mercator"What is not automatic is becoming a column of the manifest,
and so a filter in find_layer(). That takes one edit: add
the parsed key to .meta_field_map in
R/metadata.R. Adding it to .meta_required in
the same file makes check_metadata() count it towards
completeness. R/catalogue.R builds its columns from that
map and needs no change.