Skip to contents

Working with project exports

Source: vignettes/project-exports.rmd
project-exports.rmd

Introduction

Observation.org supports two primary types of data exports: classic opportunistic observations and structured project exports. Opportunistic exports consist of independent observation records. Structured project exports are designed for formal monitoring protocols: they include protocols, visits, samples, and survey effort.

When working with a project export, obs_read() remains the primary entry point. It detects the project schema automatically and returns a standard, analysis-ready obs_df (one row per observation). Because that table is observation-centric, visits or samples that yielded zero observations are not rows in this output. To analyse the full survey effort—including empty visits—obsR provides a dedicated suite of hierarchical and effort-based functions.

The examples below use a placeholder path "project.sqlite". Point them at your own Observation.org project export. Chunks are not evaluated in this vignette because a public project example is not yet shipped with the package.

Reading a project export

obs_read() enriches the standard observation table with project-specific identifiers: project, protocol, visit_id, and sample_id. Fields exclusive to classic opportunistic exports (such as uuid or validation) are NA in this context.

You can filter the import by monitoring protocol with protocol (ID or name). Standard explorers still apply to the observation-level table.

library(obsR)

# Read the entire project export
project_obs <- obs_read("project.sqlite")

# Filter by a specific protocol during import
transect_obs <- obs_read("project.sqlite", protocol = "Line Transect")

# Observation-level explorers still apply
obs_summary(transect_obs)
obs_map(transect_obs, color = "scientific_name")

Project metadata and catalogues

To understand the scope of the monitoring effort, query the project catalogues. These functions accept either a file path or a project obs_df.

  • obs_project_info(): high-level metadata and summary counts of protocols, locations, visits, and observations.
  • obs_locations(): sites, including name, geometry type, GeoJSON geom, and counts of visits, samples, and observations.
  • obs_users(): observers. observer_code is always present; observer is NA when that id is missing from the export’s user table.
info <- obs_project_info("project.sqlite")
sites <- obs_locations("project.sqlite")
observers <- obs_users("project.sqlite")

Visits and samples

Survey effort lives at the visit and sample levels. obs_visits() and obs_samples() return pivoted attribute columns using the original names from the export (no automatic translation).

Both functions have a nest argument:

  • nest = TRUE (default) keeps the hierarchy (list-columns of samples / observations), including empty children.
  • nest = FALSE returns a flat table of that level with counts only.

Filter semantics matter for effort:

  • species or group exclude visits/samples that do not contain that taxon.
  • location, user, protocol, or dates preserve empty visits/samples within the filtered scope.
# Flat visits for one site (empty visits kept)
visits_flat <- obs_visits("project.sqlite", location = "Site A", nest = FALSE)

# Nested samples (empty samples retained under their visits)
samples_nested <- obs_samples("project.sqlite", nest = TRUE)

# Visits that recorded a given species (empty visits dropped by design)
toad_visits <- obs_visits("project.sqlite", species = "Bufo", nest = FALSE)

Mapping survey sites

obs_map() plots individual observation points (lat / lng). Structured projects often need the survey coverage instead.

obs_map_visits() plots location.geom (site lines, polygons, or points). Several visits to the same site become one feature. Requires the Suggested package sf.

# Survey sites (not observation dots)
obs_map_visits("project.sqlite")

# Restrict the map with the same filters as obs_visits()
obs_map_visits("project.sqlite", protocol = "Line Transect", type = "interactive")

Effort: frequency and abundance

Project exports support effort-aware summaries. Both functions include empty visits in the denominator.

The by argument defaults to "species" and also accepts "location", "period", or combinations (for example c("location", "period")).

Filters for location, user, protocol, or dates trim the effort. Filters for species or group do not trim the denominator; they only restrict which taxa appear in the result.

The counted column

Some exports include visit.counted (NULL, n, s, i, h — the protocol “Counted” setting):

Value Meaning (short) Frequency Abundance
NULL Not assessed warning warning
n Not all species warning warning
s All species OK warning
i All species and all individuals OK OK
h Not all species; all individuals warning OK

When a visit’s value does not support the chosen analysis, obsR warns but does not drop the visit from the denominator. Current project exports store the code in visit.counted and provide the descriptions in the visit_counted catalogue. If an older export has no counted column, the functions stay silent.

# Frequency across all visits (empty visits included)
freq <- obs_frequency("project.sqlite", by = "species")

# Abundance by site and month
abund <- obs_abundance(
  "project.sqlite",
  by = c("location", "period"),
  period = "month"
)

# Taxon filter: still uses the full visit set as effort
toad_freq <- obs_frequency("project.sqlite", species = "Bufo", by = "location")

Session and transect metrics

Line-transect and GPS-track protocols store the walked path on session.geom (GeoJSON, usually a LineString), linked from samples via sample.session. This is separate from location.geom (survey site boundaries, used by obs_map_visits()) and from sample.geom (a sample-specific point or line).

obs_session_metrics() summarises each recorded session:

  • length_m — geodesic length of the track (via sf on WGS84).
  • start_lng, start_lat, end_lng, end_lat — first and last vertices of the LineString.
  • first_obs_lng, first_obs_lat — earliest observation on the primary sample (lowest sample_id linked to that session), using observation.point or lat / lng.
  • distance_to_first_obs_m — distance from the transect start to that observation.

You get one row per session_id, with visit_id, sample_id (primary sample), and n_samples (how many samples share the session). For sessions linked to more than one sample or visit, visit_ids and sample_ids retain all linked identifiers as semicolon-separated values; n_visits reports the number of linked visits. The counted column similarly retains the distinct visit.counted codes associated with the session.

Filters match obs_samples(): location, user, protocol, and dates trim which sessions appear. Unlike taxon filters on visits/samples, this function does not accept species or group. Pass a visit or sample tibble you already filtered; any extra filter arguments are then ignored.

Sessions without a valid LineString in session.geom are skipped with a warning (the rest of the export is still returned). If the SQLite file has no session table, the function warns and returns zero rows.

library(obsR)

# All sessions for one protocol
tracks <- obs_session_metrics("project.sqlite", protocol = "Line Transect")
tracks[, c("session_id", "length_m", "distance_to_first_obs_m")]

# Same filters as obs_samples()
tracks <- obs_session_metrics(
  "project.sqlite",
  location = "North Ridge",
  date = c("2022-04-01", "2022-12-31")
)

# Reuse a filtered sample table (path attached on the tibble)
samples <- obs_samples("project.sqlite", nest = FALSE)
with_session <- samples[!is.na(samples$session_id), ]
obs_session_metrics(with_session)

# Export for reporting (coordinates are plain numeric columns)
write.csv(tracks, "session_metrics.csv", row.names = FALSE)

Schema notes for session and sample.session are in vignette("export-structure") (Appendix B).

Putting it together

A typical workflow moves from scope → sites → visits → effort and transect metrics.

library(obsR)

file <- "project.sqlite"

# 1. Scope
obs_project_info(file)

# 2. Where effort was deployed
obs_map_visits(file)

# 3. Visits for one protocol (empty visits kept)
visits <- obs_visits(file, protocol = "Point Count", nest = FALSE)

# 4. Naive occupancy for that protocol
species_freq <- obs_frequency(file, protocol = "Point Count", by = "species")

# 5. Line-transect length (when sessions are recorded)
transect_metrics <- obs_session_metrics(file, protocol = "Line Transect")

# 6. Inspect
head(species_freq)
head(transect_metrics)

Next steps