Download

Documentation

1 Introductiondown-arrow
2 Setup instructionsdown-arrow
3 Frontend frameworkdown-arrow
4 Server applicationdown-arrow
5 Functionalitiesdown-arrow
6 Operating instructionsdown-arrow
7 Developing instructionsdown-arrow
8 Configuration instructionsdown-arrow
9 Enabling modulesdown-arrow
10 RPCdown-arrow
11 Application troubleshootingdown-arrow
12 Changelogdown-arrow

4 Server application

This section dives into the server functionalities.

4.1 Using oskari-server in applications

Applications can use oskari-server as dependencies with this on their pom.xml file:

<properties>
    <oskari.version>3.0.0</oskari.version>
</properties>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.oskari</groupId>
            <artifactId>oskari-server</artifactId>
            <version>${oskari.version}</version>
            <scope>import</scope>
            <type>pom</type>
        </dependency>
    </dependencies>
</dependencyManagement>

<repositories>
    <repository>
        <id>oskari_org</id>
        <name>Oskari.org release repository</name>
        <url>https://oskari.org/repository/maven/releases/</url>
    </repository>
    <repository>
        <id>oskari_org_snapshot</id>
        <name>Oskari.org snapshot repository</name>
        <url>https://oskari.org/repository/maven/snapshots/</url>
    </repository>
</repositories>

This defines:

  • Oskari version to use (so it's easy to update in one place)
  • managed dependencies used by the Oskari-version you are using to prevent accidentally using multiple versions of same dependency
  • the Maven repositories that provide pre-compiled jar-files for Maven modules in oskari-server

After this, you can use any Maven modules under oskari-server or any of its managed dependencies without a version like this:

<dependencies>
    <dependency>
        <groupId>org.oskari</groupId>
        <artifactId>servlet-map</artifactId>
    </dependency>
    ...
</dependencies>

4.2 Spring Controllers

Oskari-server uses Spring framework as a wrapper for handling HTTP-requests and security. There are two main Controllers handling most of the requests: MapController.javaand ActionRouteController.java.

Under servlet-map module

4.2.1 MapController

Handles which JSP file should be used for the application the end-user requests and sets up some information for the JSP to use for rendering the base HTML for the frontend. There are two main JSP-files that both have defaults under servlet-map: index.jsp for geoportal app and published.jsp for embedded maps.

You can override the default JSP-files with your own or add your own like shown on the sample-server-extension.

The org.oskari.spring.SpringConfig file defines the locations of the JSP files. You can define your own directory with oskari-ext.properties with oskari.server.jsp.location = /WEB-INF/jsp/ (default location for overrides) and the server first searches from that location and defaults to the /spring-map-jsp/ path in classpath if not found.

4.2.2 ActionRouteController

Handles the requests made by the frontend application. Oskari uses a concept of "action route" for processing requests made by the frontend application. Requests for action routes are processed like this:

flowchart TD
    A>User] -->|Opens page| S
    X>User] -->|Makes a search| S
    PU>User] -->|Publishes a map| S
    S[ActionRouteController] --> C{ActionRoute}
    C -->|Load page| D[GetAppSetupHandler]
    C -->|Publish a map| P[AppSetupHandler]
    C -->|Search results| E[SearchHandler]
    C -->|Application specific action| F[Your code]
    D -->|Load appsetup| AppSetupService(AppSetupService)
    P -->|Create appsetup| AppSetupService(AppSetupService)
    E -->|Search| SS(SearchService)
    SS --> |Search|OpenStreetMap>OpenStreetMap]
    SS --> |Search|WFS-service>WFS-service]
    AppSetupService --> dbId[("oskaridb")]

4.2.3 Other Controllers

StatusController

There is also a StatusController.java that can be used for load balancing/health checks.

UserRegistrationController

The control-users maven module can be used for enabling end-users to self-register to the application. It has another controller for that purpose:

https://github.com/oskariorg/oskari-server/blob/master/control-users/src/main/java/fi/nls/oskari/spring/UserRegistrationController.java

4.3 Maven modules

The backend architecture in oskari-server Maven-modules can be divided into three layers: service layer, control layer and interface layer:

  1. The interface layer is very light with Spring framework Controllers for handling requests and can be easily substituted to run as portlets or similar.
  2. The controllers pass concrete HTTP-requests on to Oskari control-modules that can further process the requests and write responses.
  3. Services are used by the control-modules to handle business-logic. The service-modules could (in theory) be used in any Java-based software as libraries.

Service layer

Service modules should be common libraries usable in any application. The actual business logic for Oskari operations should be in these modules.

  • service-base has some common helpers and domain classes which are used throughout Oskari backend
  • service-permission is a generic authorization lib for deciding who gets to see/do what
  • service-search is a generic search lib that can be extended by adding and registering search channels.
  • service-map has most (maybe a bit too much) of the business logic for servicing the Oskari functionalities
  • service-control defines the control/routing structures/interfaces for control-layer to build upon
  • shared-test-resources has some common helpers/templates to help testing

Control layer

Control modules build on top of the service layer.

  • control-base is the basis for all control-modules and has most of the basic request handlers needed by the Oskari frontend.
    • NOTE! control-base contains some very specific functionalities that should be separated into separate control-extensions (for example thematic maps support)
  • control-myplaces provides funtionality related to myplaces functionality.
  • control-example provides example implementations for functionalities required by Oskari but usually overridden by platform specific functionalities such content management for user guide etc.
  • content-resources has tools, templates and scripts for populating and migrating the database

Functions:

  • Handles requests made by Oskari frontend
  • Parses request parameters for input values to be used on service invocations
  • calls one or more services to do business logic
  • format a response based on service response

Interface layer

The interface-modules build on top of the control-modules. Basically an HTTP interface with reference implementations for:

  • HTTP Servlet: oskari-server/servlet-map
  • Webapp: sample-server-extension/webapp-map

Responsible for:

  • Handling user sessions
  • Generating an ActionParameters object based on incoming request abstracting/normalizing the request for control layer
  • Forwarding the request to control layer

4.4 Database overview

This section should give an overview of the database structure.

4.5 Database migration

This section should give information about database migrations and Flyway-modules.

4.5.1 Upgrading

When upgrading Oskari version the database schema might require upgrading as well. The upgrading and initialization of an empty database is done automatically with migration scripts using https://flywaydb.org/. This document describes the procedure and provides insight for debugging on possible error scenarios.

The migration happens when the webapp is started. The software version is recorded into the database and each upgrade script that is tagged/named with a later version than have been run before is executed in version sequence on startup and the current database status is updated so the next startup won't trigger the same migrations.

The upgrade is done with a modular setup with these default modules:

  • oskari: this is the base database for Oskari
  • myplaces: these are tables/triggers etc for the my places functionality
  • userlayer: these are tables/triggers etc for the data importing functionality

Oskari-based server-applications can also use a migration module to initialize the database content and applications setups. For example the sample-server-extension uses the app module to initialize a couple dataproviders, default users, an application specific bundle, application setups (collection of bundles to be shown as geoportal for the user) and layers to the database as content.

The default non-core module settings are defined in oskari.properties and can be overridden in oskari-ext.properties:

# mark any addition module property tokens so we can check/add them automatically
db.additional.modules = myplaces, userlayer

Each module keeps track of it's state using a database table named oskari_status_[module]. The core database status is tracked in table named oskari_status. Each module can also have their own datasource providing the database connections. By default a common datasource is used that is configured in oskari-ext.properties file:

db.url=jdbc:postgresql://localhost:5432/oskaridb

The built-in migration scripts can be found in Github:

For the sample-server-extension they are in the application repository:

4.6 Troubleshooting

Oskari logs a message (with log level info) "Oskari-map checking DB status" when the migration starts. When the core database has been migrated successfully, you should see a message "Oskari core DB migrated successfully". After this any configured modules are migrated. After each module migration the log should show a success message for that module ("[module] DB migrated successfully"). If the core database migration fails for any reason, the message "DB migration for Oskari core failed!" is logged with stacktrace detailing the error. The same is true with any module with message "DB migration for module [module] failed!". Problems need to be addressed case-by-case so the stack trace on error is the only tool that can be provided.

Commonly known issues are listed in Writing upgrade scripts document. If any error occurs the migration will stop and the database status can be seen on the database status table for the module in question. On next startup the migration will resume from the version that caused the error.

The database tables show migrations that have been run successfully, for example oskari_status_userlayer looks like this:

version_rank installed_rank version description type script checksum installed_by installed_on execution_time success
1 1 0.1 << Flyway Baseline >> BASELINE << Flyway Baseline >> postgres 2015-06-25 15:25:05.949 0 t
2 2 1.0 create-tables SQL V1_0\__create-tables.sql 1112871463 postgres 2015-06-25 16:29:18.299 40 t
3 3 1.0.1 create triggers SQL V1_0_1\__create_triggers.sql 668818731 postgres 2015-06-25 16:29:18.354 26 t
4 4 1.0.2 Insert id for userlayerdata JDBC flyway.userlayer.V1_0_2\__Insert_id_for_userlayerdata postgres 2015-06-25 16:53:00.717 4 t

This shows that:

  • the database has been initialized with version 0.1
  • SQL script V1_0__create-tables.sql has been run that by description creates tables - bringing the database version to 1.0
  • SQL script V1_0_1__create_triggers.sql has been run that by description creates database triggers - bringing the database version to 1.0.1
  • Java upgrade has been performed that by descriptions inserts some id for userlayerdata - resulting the database version 1.0.2

The next script that updates the userlayer module needs to be named with a bigger version number for example V1_0_3__upgrade.sql or V1_1__upgrade.sql. Versions can technically skip from 1.0 to 37.2.26 with no issues, but the version should follow the software version as convention.

4.6.1 Note! Database locking

When flyway begins the migration it starts a database transaction that locks the status table. If the migration takes a long time or is broken and you decide to kill the server before the migration is done the status table lock will remain. This kind of lock can be removed manually like this in Postgres 9.3:

  1. Create a view that lists the locked tables
CREATE VIEW blocking_procs AS
SELECT
    kl.pid as blocking_pid,
    ka.usename as blocking_user,
    ka.query as blocking_query,
    bl.pid as blocked_pid,
    a.usename as blocked_user,
    a.query as blocked_query,
    to_char(age(now(), a.query_start),'HH24h:MIm:SSs') as age
FROM pg_catalog.pg_locks bl
    JOIN pg_catalog.pg_stat_activity a
    ON bl.pid = a.pid
    JOIN pg_catalog.pg_locks kl
    ON bl.locktype = kl.locktype
    and bl.database is not distinct from kl.database
    and bl.relation is not distinct from kl.relation
    and bl.page is not distinct from kl.page
    and bl.tuple is not distinct from kl.tuple
    and bl.virtualxid is not distinct from kl.virtualxid
    and bl.transactionid is not distinct from kl.transactionid
    and bl.classid is not distinct from kl.classid
    and bl.objid is not distinct from kl.objid
    and bl.objsubid is not distinct from kl.objsubid
    and bl.pid <> kl.pid
    JOIN pg_catalog.pg_stat_activity ka
    ON kl.pid = ka.pid
WHERE kl.granted and not bl.granted
ORDER BY a.query_start;
  1. List the locks
SELECT * FROM blocking_procs;
  1. Kill the process that is holding the lock:
SELECT pg_terminate_backend([blocking_pid]);

When your server starts and logs the "checking DB status" message and seems stuck there you might have this kind of lock in the database. If you see the "migrated successfully" message for each module, then it's something else.

4.7 Advanced: upgrade application specific database/data

Any customized application can setup the automatic migration by adding some configurations in oskari-ext.properties and providing the application specific upgrade scripts. Applications are treated as modules that can opt-in on using the automatic upgrade by defining a module in oskari-ext.properties (here we add a module called myapplication):

db.additional.modules = myplaces, userlayer, myapplication

By default a modules upgrade scripts are discovered from classpath in location /flyway/[module]. This includes both SQL and Java upgrade-files so for Java the package needs to be flyway.[module]. If you want to change the default script location or provide alternatives, you can specify a comma-separated list in oskari-ext.properties:

db.myapplication.script.locations = /flyway/myapplication, /upgrade/scripts/in/here/also

Note! If you define a custom module it needs to have at least one upgrade script. Otherwise Flyway will log it as an error. Instructions for writing scripts can be found in the Writing upgrade scripts document.

If the application uses another datasource than the default you need to configure it with oskari-ext.properties:

db.myapplication.jndi.name=jdbc/MyApplicationDS
db.myapplication.url=[db url]
db.myapplication.username=[db user]
db.myapplication.password=[db pass]

If you want to use another table name than oskari_status_[module] you can override it with this property:

db.myapplication.status_table=my_status_table_name

4.7.1 Examples

Arctic-SDI geoportal

In the Arctic-SDI geoportal, myplaces or userlayers are not needed so oskari-ext.properties defines only the application specific module. This means that the database structure for myplaces and userlayers are not created when starting the application:

db.additional.modules=asdi

Modules can be added when the functionality is needed so when myplaces functionality is added to the application the properties can be updated and on the next startup the myplaces database will be initialized and migrated up to date:

db.additional.modules=asdi, myplaces

Arctic-SDI geoportal also has an example for modifying the appsetups when a new functionality (frontend bundle) is added or zoomlevels are tuned: https://github.com/arctic-sdi/oskari-server-extensions/tree/develop/asdi-resources/src/main/java/flyway

4.8 Writing migration scripts

4.9 DONT's

  • Don't modify an existing script once it's committed, flyway calculates checksum for scripts so any change in existing script will result in a migration error.
  • Don't link tables between migration modules. Modules are considered to be standalone and constraints between them are likely to cause problems.
  • Be very careful when doing updates to database content. Other than registering new bundles to portti_bundle almost any content-related updated should be done in an application module. F.ex. adding bundles to views etc are application specific updates.

4.10 DO's

  • Make sure the upgrade scripts have proper error handling. Database environments may vary as the content in the database. It's always better to check things instead of assuming.
  • Keep upgrade scripts small. The smaller they are the less can go wrong.

4.11 Script naming

File naming is important since migration scripts are located and versioned by naming conventions:

  • the names should start with capital V
  • followed by version number that is greater than any existing script version and based on the Oskari/module/application version
  • version and description is separated by double-underscore __
  • description
  • for SQL scripts the file extension is .sql and saved in a folder /flyway/[module]
  • for Java upgrades the package needs to be flyway.[module] where core Oskari upgrades are flyway.oskari

In a basic maven setup this will result in files like these:

  • oskari-server/content-resouces/src/main/resources/flyway/oskari/V1_31_0__add_mycolumn_for_mytable.sql
  • oskari-server/content-resouces/src/main/java/flyway/oskari/V1_31_1__populate_mycolumn_from_CSV.java

Existing scripts can be found in Github:

Application-specific upgrade example:

Note the script locations can be changed by configuration as described in the Upgrading document.

For more information:

4.12 Main configuration file

This section should give information about oskari(-ext).properties.

4.13 Authentication

This section should give information about Spring framework configurations and what applications can do to provide customized configurations.

4.14 Authentication and user management (requires backend)

Oskari has bundles for user management and for management of layers' priviledges. More information about the bundles will be added to this documentation later.

4.14.1 SSO authentication support

Oskari supports authentication and implementing/hooking into different authentication solutions.