3 Setup instructions

This section contains instructions for setting up a new Oskari instance.

C4Deployment

Person(user, "User", "")

Boundary(service, "Oskari-based service", ""){
    Boundary(frontend, "Web server", "nginx/httpd"){
        Component(sample-app, "sample-application", "Javascript")
    }
    Boundary(server, "Servlet container", "tomcat/jetty"){
            Component(sample-server, "sample-server-extension", "Java")

    }
    Boundary(db, "Databases", ""){
        ContainerDb(postgres, "PostgreSQL", "PostGIS/SQL", "Application config and user data")
        ContainerDb(redis, "Redis", "", "Caching")
    }
}
System_Ext(sdi, "OGC and other APIs", "External services")


Rel(user, sample-app, "", "")
Rel(sample-app, sample-server, "", "")
Rel(sample-server, postgres, "", "")
Rel(sample-server, redis, "", "")
Rel(sample-server, sdi, "", "")
Rel(user, sdi, "", "")

3.1 Setup database

This document describes how to set up a single database for the oskari-server component.

The standalone Oskari server depends on the availability of PostgreSQL with PostGIS extension for serving content and authenticating users.

3.1.1 Database requirements:

The following components are assumed pre-installed:

• PostgreSQL 11+ (Known to work with 11, 12, 14 mainly affected by the FlywayDB version we use. Developed using 14.)
• PostGIS (Developed using 3)

3.1.2 Steps to create the database

Create an empty database with PostGIS extension

The default configurations assume the database name is "oskaridb". It's configurable in oskari-ext.properties. Run the create database SQL in for example psql or pgAdmin:

 CREATE DATABASE oskaridb
 WITH OWNER = postgres
   ENCODING = 'UTF8'
   TABLESPACE = pg_default
   CONNECTION LIMIT = -1;

Add the PostGIS extension (for the oskaridb database) by first connecting to the database (example for psql):

\c oskaridb

Add the extension by running SQL:

CREATE EXTENSION postgis;

Setup a database user for oskaridb

Run these commands to create default user with all privileges

CREATE USER oskari WITH PASSWORD 'oskari';
GRANT ALL PRIVILEGES ON DATABASE oskaridb to oskari;

The preconfigured user in Oskari Jetty-bundle is oskari with the password oskari. See Setup Jetty documentation for details where changes are needed when using another database user.

Application initialization and database content

The empty database will be populated when the oskari-server is started for the first time. The database population is split into modules. The core module creates and migrates the main database tables used by Oskari. The default configuration has a module named "example" enabled that will populate an example app and maplayer to get a nice unboxing experience. You should replace "example" module with your own app configuration and content for anything other than basic development and playing around.

To learn how to customize Oskari including populating the database with your own content instead of example content see:

• Create a custom Oskari-server extension

3.2 Setup Jetty

This section contains instruction on setting up an instance from a downloaded jetty package with pre-installed/configured Oskari

For setting up an oskari instance from source code skip this section and move to Setup development environment

After this you will have Oskari running including

• Oskari-frontend based sample-application (https://github.com/oskariorg/sample-application)
• Oskari-server based sample webapp (https://github.com/oskariorg/sample-server-extension)

Requirements

The following are required for setting up Jetty.

• JDK 8
• Database available: Instructions for setting up database

Setting up Jetty

Follow the steps below to get Jetty properly set up.

1) Download the Jetty Bundle

2) Unpack the zip file to selected location

The zip includes Howto.md, jetty-distribution-9.4.12.v20180830 (referred as {jetty.home}) and oskari-server folder (referred as {jetty.base})

3) Configure the database properties (host/credentials) by editing {jetty.base}/resources/oskari-ext.properties

db.url=jdbc:postgresql://[host]:[port]/[dbname]
db.username=[user]
db.password=[passwd]

4) Startup the Jetty by running (in {jetty.base})

java -jar ../jetty-distribution-9.4.12.v20180830/start.jar

Note that for folder references it's important where you run the command/what is the working directory so run the command in oskari-server folder and refer to start.jar under the {jetty.home}:

5) After Jetty is up and running open a browser with URL

http://localhost:8080

You can login as:

• user with username "user" and password "user"
• admin with username "admin" and password "oskari"

Advanced configuration

For further configuration check out Advanced jetty configuration

3.3 Setup Oskari development environment

This document describes how to setup development environment for Oskari using source code. If you want to develop / maintain oskari yourself this is the correct section. For setting up an instance using the ready jetty bundle check out Setup Jetty

3.3.1 Requirements

The following items are assumed installed.

• JDK 8
• Maven 3+ (developed using 3.5.0)
• Git client (http://git-scm.com/) - Optionally download zip file from https://github.com/oskariorg/oskari-server
• {jetty.base} refers to the oskari-server folder in an unzipped Jetty bundle

3.3.2 Setup Git configuration

Configure line endings: https://help.github.com/articles/dealing-with-line-endings/

Ignore file permissions:

git config --global core.fileMode false

3.3.3 Fetch Oskari-server source code

With commandline git:

git clone https://github.com/oskariorg/oskari-server.git

Note! You can also download the codes in zip format from Github, but for contributing any changes to Oskari git is mandatory. Additional Maven modules can be contributed outside git though if they are compatible with the current develop/master branch, but this is not adviced.

Note! The sample application, including its source code and build, is available in {jetty.base}/sample-application within the Jetty bundle. The Jetty bundle uses the pre-built code located in the dist folder. To update the sample application you can replace the contents of the sample-application folder with a clone from https://github.com/oskariorg/sample-application, and build the application following the instructions in its README.md file.

3.3.4 Build Oskari server

This will build all modules that Oskari server is composed of.

cd oskari-server
mvn clean install

3.3.5 Fetch sample-server-extension source code and compile it

To test your changes on a running web app you can use sample-server-extension to create a webapp using your modified version of oskari-server. Check that the oskari.version in pom.xml matches the project version of oskari-server you built.

git clone https://github.com/oskariorg/sample-server-extension.git
cd sample-server-extension
mvn clean install

3.3.6 Copy updated/relevant artifacts under `{jetty.base}/webapps`

Map functionality: sample-server-extension/webapp-map/target/oskari-map.war

3.4 Updating Oskari version

Note! We are updating the frontend and server versions at the same time and for best compatibility it's best to use same version in both. Hotfix/patch versions might not always match but with version major.minor.patch the major and minor versions should match.

3.4.1 Migration and release notes

Simply updating the version of Oskari dependency for the frontend and server and building new versions of your app/server is enough in most cases. But sometimes there are additional things that need to be done when updating. For example your application code might not compile after updating if we have changed some class in oskari-server that you are using directly in your app or there might be some things that should be done while updating that are not as obvious. These things are documented on the Migration Guide in oskari-server repository. You should also at least skim through the Release notes for both server and frontend to see if there are something that might affect your app. There might be a new implementation of a functionality/bundle that you want to use instead of the old one or things like this.

So take a look at these before updating:

• Migration Guide
• Server release notes
• Frontend release notes

It's also a good idea to see any PRs/changes for the sample application to see if there are things that you can streamline/change in your application for the newer version:

• https://github.com/oskariorg/sample-application
• https://github.com/oskariorg/sample-server-extension

Note! The idea is not that you use the sample-application/server directly but use it as a template to create your own app/server. For example the database migrations in server-template assume they are run on an empty database and might not work properly if copied directly to your server code. We change the sample migrations between versions to make the most sense as simple examples for the latest version while actual apps (copies of sample-server) should work to migrate the existing database of the app.

3.4.2 Server

Updating an Oskari-powered server that is based on the sample-server-extension template is done by updating the value of oskari.version property in pom.xml file:

    
        1.55.1
        ...
    

To the new version:

    
        1.56.0
        ...
    

After this run mvn clean install to generate a new oskari-map.war under [your-server-repository-root]/webapp-map/target. Replace the oskari-map.war under the oskari-server/webapps folder (in a setup similar to the Oskari/Jetty download package) with the new one.

3.4.3 Frontend

Updating an Oskari application that is based on the sample-application template is fairly simple as well. For the frontend the oskari-frontend (and possible oskari-frontend-contrib) dependency is updated by changing the version number in package.json file from the current version in the app:

  "dependencies": {
    "oskari-frontend": "https://git@github.com/oskariorg/oskari-frontend.git#1.55.2"
  },

To the new version:

  "dependencies": {
    "oskari-frontend": "https://git@github.com/oskariorg/oskari-frontend.git#1.56.0"
  },

When updating Oskari version you might also want to change the application version in package.json on your application to signal that the application has been updated.

After this you need to run npm install to install any new/changed libraries and npm run build to generate a new build under [your-application-repository-root]/dist/[version on package.json]. After the build the application repository can be served as static resources from Jetty/nginx or similar server.

Note! To use the newly built frontend you will need to update oskari-ext.properties under oskari-server/resources to link the new frontend version:

  oskari.client.version=dist/[version from your package.json]

Building the frontend for production

Modern web applications are developed using JavaScript syntax that is not fully supported by older browsers and the code needs to be processed before it can be used by end-user browsers. This step also includes bundling and minifying the code so it's more compact for consumption than the human-friendly version that is used for development. This will reduce the startup time for the end-user dramatically.

With the default sample application (a pre-built version is included in the zip download) you can run npm run build to generate the minified Javascript files. This will result in folders and files generated under [your frontend repository root]/dist/.

The build works by reading a main.js file that links all the functionality/bundles you want to use in your application together and should match the appsetup (bundle collection) used on the website you are creating including any dynamic (role-based) bundles that are added on the fly. So basically all the bundles you want to use in that application. Here's an example for the basic geoportal appsetup and for an embedded map. These are processed when running npm run build.

If you take a look at the package.json script for build you can see that the parameter --env.appdef=applications is used to point the build to search for application main.js files under the applications folder. You can change this in your own app as you wish.

The best point to start customizing your app is changing the main.js file under the sample geoportal application to only include the bundles that you are using. This reduces the amount of code the end-user needs to load. You can also safely remove the 3D applications if you don't need them. The application for embedded/published maps usually stays more or less the same and the geoportal one is usually customized based on requirements.

When changing main.js to include application specific bundles and/or customizing the application you need to run npm run build again to create new minified result. You will also need to do this when updating the new version of Oskari or after changing any application specific code/bundles you want to use.

Developing the application specific code

While developing your application it's recommended to run npm run start instead of npm run build over and over again. The start script launches a Webpack dev-server for the frontend that will process the code while you develop it. The dev-server starts at port 8081 and assumes the normal backend/server is running on the default port 8080 (The oskari-server based server component). The dev-server also assumes that the client version on the server is configured in oskari-ext.properties under oskari-server/resources as:

  oskari.client.version=dist/devapp

3.5 Advanced configuration

The section contains advanced setup and configuration instructions which users usually do not need to worry about.

3.5.1 Advanced Jetty configuration

Jetty default configuration

The preconfigured Jetty uses these defaults. These can be changed by modifying {jetty.base}/resources/oskari-ext.properties.

Redis:

• redis running on localhost at default port (6379)

Database (Postgres with postgis extension)

• db URL: localhost in default port (5432)
• db name: oskaridb
• db user: oskari/oskari

Oskari (provided in Jetty bundle)

• url: http://localhost:8080/

Custom configurations

1) Removing the unnecessary parts

Oskari-server can run with just the oskari-map webapp. If you don't need all the features, you can remove them from under {jetty.base}/webapps.

You will also need to remove the corresponding parts of the UI so users don't have access to them. This is done by removing "bundles" from "appsetups" (these are Oskari concepts: bundles provide functionalities and appsetup defines which bundles are used in your app) and currently it needs to be done by modifying the database content. Bundles are linked to appsetups in the database table portti_view_bundle_seq and functionalities are removed from the UI by deleting rows from the table.

2) Editing article content

• User guide: edit the file in {jetty.base}/resources/articlesByTag/userguide.html
• Publisher terms of use: edit the file in {jetty.base}/resources/articlesByTag/termsofuse__mappublication__en.html

3) Changing the default port**

• provide port in command line:

  java -jar ${jetty.home}/start.jar jetty.http.port=8080

• change {jetty.base}/resources/oskari-ext.properties where ever 8080 is referenced

4) Proxy settings

If you need a proxy to access internet you can configure it in {jetty.base}/start.d/oskari.ini

-Dhttp.proxyHost=
-Dhttp.proxyPort=
-Dhttp.nonProxyHosts=
-Dhttps.proxyHost=
-Dhttps.proxyPort=
-Dhttps.nonProxyHosts=

5) Database url/name/user/pass are changed {jetty.base}/resources/oskari-ext.properties needs to be updated

db.url=jdbc:postgresql://[host]:[port]/[dbname]
db.username=[user]
db.password=[passwd]

5) Using external Redis {jetty.base}/resources/oskari-ext.properties needs to be updated

redis.hostname=localhost
redis.port=6379
redis.pool.size=10

6) How the Jetty bundle was built

See the Howto.md inside the zip-file for details

3.5.2 Setup Redis for Oskari (Optional)

Redis is used for caching data for example from statistical datasources to provide a cleaner user experience for statistical map functionalities (optional part of oskari-map webapp).

1) Get Redis

Install binaries from http://redis.io/ or from your platforms package repository.

2) Install

And startup redis-server. The default port Redis listens to is 6379.

3). Configure Oskari (Optional)

Oskari expects Redis to be found in the default port (6379) on the same server as Oskari ("localhost"). If you have it running on another host/port you need to change the oskari-ext.properties:

redis.hostname=localhost
redis.port=6379

3.5.3 Configuring nginx (Optional)

This guide gives an example for configuring a reverse proxy for Oskari-server using nginx. The latest nginx version at the time of writing is 1.8.1 which has been used to test these configurations.

Using a reverse proxy is not required for development, but is recommended for production use

You can find example configurations in https://github.com/oskariorg/sample-configs/tree/master/nginx

In /etc/nginx/nginx.conf turn on gzip support

gzip  on;

Most of the other configurations can be done in /etc/nginx/conf.d/default.conf

Assumptions

Oskari frontend code.

Oskari frontend code should be made available in the server directory /opt/public/oskari. This can be changed by modifying these lines:

    root /opt/public/;

    # Oskari frontend files
    location ^~ /Oskari/ {
        rewrite ^/Oskari/(.*)$ $1 break;
        try_files /oskari/$1 oskari/$1/ =404;
    }

Oskari-server

Oskari server should be running on localhost in port 8080. This can be changed by modifying these lines:

upstream oskariserver {
    server localhost:8080;
}

Cross-site request forgery protection

Java libraries don't support the SameSite-flag for cookies yet so we need to protect our service from session manipulation and CSRF by modifying the cookies on nginx. This snippet modifies the cookie Jetty gives and adds secure, httponly and samesite-flags on it. SameSite-flag means that browsers don't send for example the session cookie when requests originate from a different domain.

    # Oskari-server Jetty location
    location / {
        ...

        # set all cookies to secure, httponly and samesite by modifying "path"
        proxy_cookie_path / "/; secure; HttpOnly; SameSite=lax";
    }

HTTPS-configuration

The following enables HTTPS on the server. Add the certificates on:

• /etc/nginx/ssl/public.crt for public key
• /etc/nginx/ssl/private.rsa for private key

or change the configuration accordingly.

    # ssl config - optional, but recommended for offering https-urls
    listen       443 ssl;
    ssl_certificate /etc/nginx/ssl/public.crt;
    ssl_certificate_key /etc/nginx/ssl/private.rsa;

    # ssl security settings - optional, but recommended
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
    server_tokens off;
    # /ssl config

Proxying requests to non-secure services

Oskari-server will proxy any map layer services that are not secure (https) to allow browsers show the map on secure domain. This requires the application server to be aware if the incoming request is secure. In Jetty this can be accomplished by forwarding some headers to Jetty hosting oskari-map:

    location / {
        ...
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        ...
    }

The Jetty must also be configured to be aware that it has a reverse-proxy. This is done by modifying {jetty.base}/start.d/oskari.ini:

# ---------------------------------------
# Module: http-forwarded
# Enable X-Forwarded-For headers handling
# Needed for:
# - Webpack frontend development
# - forward proxy on production (for example nginx -> jetty) where the proxy is handling TLS etc
# ---------------------------------------
--module=http-forwarded

This enables the http-forwarded module for Jetty 9 which allows webapps like Oskari-server get additional information about the requests it receives. This allows Oskari to proxy requests that don't support HTTPS.

Note! This also adds a great amount of network traffic going through your server!

3.5.4 Server clustering (Optional)

Oskari server can be used in a clustered environment setup. A server cluster can be used to increase availability, reliability and scalability and help deal with software crashes/hardware failures etc in the system.

A simple clustered environment can be set up by having 1-n Jetty-servers (or Tomcat etc) running Oskari-server based webapp and having a load balancer like nginx as a an entrypoint that forwards requests to the Jetty-instances. The requests can be forwarded in a round-robin fashion. "Sticky sessions" are not required but could benefit debugging etc (see Identified challenges).

Currently this means:

• User sessions can be saved/tracked in Redis (shared between nodes)
• Caches communicate removals/flushes between cluster nodes
• Programmatically setting log level is communicated between cluster nodes

Note! A "cluster" with just one server instance still means persistent sessions (sessions don't "die" with server restart etc) so it's worth considering.

Enabling cluster handling

Clustered environment handling is enabled with oskari.profiles in oskari-ext.properties:

# Comma-separated list of profiles to activate.
oskari.profiles=redis-session

Add redis-session to the profiles list

Code examples for adding more cluster aware functionality

Check status

Check if we have cluster handling enabled:

boolean isClustered = org.oskari.cluster.ClusterManager.isClustered();

Messaging

Calling ClusterManager.getClientFor([functionality id]) returns a ClusterClient instance that is created on first call and shared between returning calls. This keeps the number of Redis-connections manageable. The ClusterClient can be used to send messages and listen to messages. The ClusterManager handles that messages from the same instance are not received by listeners of that same instance. ClusterClient handles that messages from other functionalities on the same instance are not received by other functionalities for example "cache" functionality doesn't receive messages from "logger" functionality making it easier to handle messages based on the functionality.

For example cache is used by Oskari cache classes as functionality id.

Messaging also has a channel that can be used to further filter messages for specific listeners. Channel could be considered an event name for usual event based code. In the cache functionality channel is used to identify the cache instance (maplayer, dataprovider etc).

Sending messages

String functionalityId = "cache";
String channel = "maplayer";
String msg = "REM:1";
ClusterManager
    .getClientFor(functionalityId)
    .sendMessage(channel, msg);

This would send a message REM:1 to cache functionality with channel maplayer. This would signal other cluster nodes that cache with name maplayer should be notified with the message REM:1. The functionality that is doing messaging is responsible for implementing the message parsing/protocol ie. what kind of message signifies what in the functionality.

Listening to messages

Listen to messages from other cluster nodes based on functionality id and more specific message type channel. The code below can be used to listen to messages from other cluster nodes that have been sent with the code snippet above.

String functionalityId = "cache";
String channel = "maplayer";
ClusterManager
    .getClientFor(functionalityId)
    .addListener(channel, (msg) -> handleClusterMsg(msg));

The handleClusterMsg() will receive the REM:1 message from the above example and in the case of caches will remove the value with key 1 from the cache. Note that channel is used to identify the cache so only the maplayer cache would remove the item with key 1, not for example a cache for dataproviders.

Identified challenges

Server clusters usually present some challenges for servers environments as well. Here's something we've identified:

Logging

Logging in clustered environment means that a single page view by a single user can generate log messages on different servers (different log files). This requires an instance specific solution like:

• "sticky session" where the load balancer in front of Oskari-server forwards requests from single user to the same server instance.
• a centralized logging system where the log from each node is gathered on a central system.

The centralized logging would also benefit from a "request id" that could be used to group log messages/user without identifying the user details. This hasn't been implemented as it is not trivial and would require passing some identifier to different parts of the system that don't require it now.