3 Setup instructions
This section contains instructions for setting up a new Oskari instance.
C4Deployment
Person(user, "User", "")
Boundary(service, "Oskari-based service", ""){
Boundary(frontend, "Web server", "nginx/httpd"){
Component(sample-app, "sample-application", "Javascript")
}
Boundary(server, "Servlet container", "tomcat/jetty"){
Component(sample-server, "sample-server-extension", "Java")
}
Boundary(db, "Databases", ""){
ContainerDb(postgres, "PostgreSQL", "PostGIS/SQL", "Application config and user data")
ContainerDb(redis, "Redis", "", "Caching")
}
}
System_Ext(sdi, "OGC and other APIs", "External services")
Rel(user, sample-app, "", "")
Rel(sample-app, sample-server, "", "")
Rel(sample-server, postgres, "", "")
Rel(sample-server, redis, "", "")
Rel(sample-server, sdi, "", "")
Rel(user, sdi, "", "")3.1 Setup database
This document describes how to set up a single database for the oskari-server component.
The standalone Oskari server depends on the availability of PostgreSQL with PostGIS extension for serving content and authenticating users.
3.1.1 Database requirements:
The following components are assumed pre-installed:
- PostgreSQL 11+ (Known to work with 11, 12, 14 mainly affected by the FlywayDB version we use. Developed using 14.)
- PostGIS (Developed using 3)
3.1.2 Steps to create the database
Create an empty database with PostGIS extension
The default configurations assume the database name is "oskaridb". It's configurable in oskari-ext.properties. Run the create database SQL in for example psql or pgAdmin:
CREATE DATABASE oskaridb
WITH OWNER = postgres
ENCODING = 'UTF8'
TABLESPACE = pg_default
CONNECTION LIMIT = -1;Add the PostGIS extension (for the oskaridb database) by first connecting to the database (example for psql):
\c oskaridbAdd the extension by running SQL:
CREATE EXTENSION postgis;Setup a database user for oskaridb
Run these commands to create default user with all privileges
CREATE USER oskari WITH PASSWORD 'oskari';
GRANT ALL PRIVILEGES ON DATABASE oskaridb to oskari;The preconfigured user in Oskari Jetty-bundle is oskari with the password oskari. See Setup Jetty documentation for details where changes are needed when using another database user.
Application initialization and database content
The empty database will be populated when the oskari-server is started for the first time. The database population is split into modules. The core module creates and migrates the main database tables used by Oskari. The default configuration has a module named "example" enabled that will populate an example app and maplayer to get a nice unboxing experience. You should replace "example" module with your own app configuration and content for anything other than basic development and playing around.
To learn how to customize Oskari including populating the database with your own content instead of example content see:
3.2 Setup Jetty
This section contains instruction on setting up an instance from a downloaded jetty package with pre-installed/configured Oskari
For setting up an oskari instance from source code skip this section and move to Setup development environment
After this you will have Oskari running including
- Oskari-frontend based sample-application (https://github.com/oskariorg/sample-application)
- Oskari-server based sample webapp (https://github.com/oskariorg/sample-server-extension)
Requirements
The following are required for setting up Jetty.
- JDK 8
- Database available: Instructions for setting up database
Setting up Jetty
Follow the steps below to get Jetty properly set up.
1) Download the Jetty Bundle
2) Unpack the zip file to selected location
The zip includes Howto.md, jetty-distribution-9.4.12.v20180830 (referred as {jetty.home}) and oskari-server folder (referred as {jetty.base})
3) Configure the database properties (host/credentials) by editing {jetty.base}/resources/oskari-ext.properties
db.url=jdbc:postgresql://[host]:[port]/[dbname]
db.username=[user]
db.password=[passwd]4) Startup the Jetty by running (in {jetty.base})
java -jar ../jetty-distribution-9.4.12.v20180830/start.jarNote that for folder references it's important where you run the command/what is the working directory so run the command in oskari-server folder and refer to start.jar under the {jetty.home}:
5) After Jetty is up and running open a browser with URL
http://localhost:8080You can login as:
- user with username "user" and password "user"
- admin with username "admin" and password "oskari"
Advanced configuration
For further configuration check out Advanced jetty configuration
3.3 Setup Oskari development environment
This section describes how to setup development environment for Oskari using source code. If you want to customize Oskari, this is the correct section.
For setting up an instance using the ready Jetty bundle see Setup Jetty
3.3.1 Requirements
The following items are required for the development process:
- JDK 8
- Maven 3+ (developed using 3.5.0)
- Git client (git is mandatory for contributing any changes to Oskari) - Optionally download zip file from oskari-server GitHub repo
{jetty.base}refers to the oskari-server folder in an unzipped Jetty bundle
3.3.2 Setup Git configuration
Configure line endings: https://help.github.com/articles/dealing-with-line-endings/
Ignore file permissions:
git config --global core.fileMode false3.3.3 Fetch Oskari-server source code
With command line git:
git clone https://github.com/oskariorg/oskari-server.gitOr download the codes in zip format from GitHub.
Additional Maven modules can be contributed outside git though if they are compatible with the current develop/master branch, but this is not advised.
Note! The sample application, including its source code and build, is available in {jetty.base}/sample-application within the Jetty bundle. The Jetty bundle uses the pre-built code located in the dist folder. To update the sample application you can replace the contents of the sample-application folder with a clone from https://github.com/oskariorg/sample-application, and build the application following the instructions in its README.md file.
3.3.4 Build Oskari server
Build all modules that Oskari server is composed of with the following command:
cd oskari-server
mvn clean install3.3.5 Fetch sample-server-extension source code and compile it
To test your changes on a running web app you can use sample-server-extension to create a web app using your modified version of oskari-server.
Check that the oskari.version in pom.xml matches the project version of oskari-server you built.
Download and install the sample-server-extension with the following commands:
git clone https://github.com/oskariorg/sample-server-extension.git
cd sample-server-extension
mvn clean install3.3.6 Copy updated/relevant artifacts under `{jetty.base}/webapps`
Map functionality: sample-server-extension/webapp-map/target/oskari-map.war
3.4 Updating Oskari version
This section guides you through updating your Oskari instance.
We are updating the frontend and server versions at the same time and for the best compatibility it's best to use same version in both. Hotfix/patch versions might not always match but with version major.minor.patch the major and minor versions should match.
Before updating, the following links are recommended to at least skim through:
3.4.1 Migration and release notes
In most cases updating Oskari version means:
- updating the version of Oskari dependency for the frontend and server and
- building new versions of your app/server
Sometimes there are additional things that need to be done when updating. These are documented on The Migration Guide in oskari-server repository.
There can be also application spesific things that need to be taken into account when updating. For example, your application code might not compile after updating if you have changed some class in oskari-server that you are using directly in your app.
You should also at least skim through the Release notes for both the server and the frontend to see if there is something that might affect your app. There might be, for example, a new implementation of a bundle/functionality that you want to use instead of the old one.
It's also a good idea to see any PRs/changes for the sample application to see if there are things that you can streamline/change in your application for the newer version:
- https://github.com/oskariorg/sample-application
- https://github.com/oskariorg/sample-server-extension
Note! The idea is not that you use the sample-application/server directly but use it as a template to create your own app/server. For example the database migrations in server-template assume they are run on an empty database and might not work properly if copied directly to your server code. We change the sample migrations between versions to make the most sense as simple examples for the latest version while actual apps (copies of sample-server) should work to migrate the existing database of the app.
3.4.2 Updating the server
Updating an Oskari-powered server that is based on the sample-server-extension template is done by updating the value of oskari.version property in pom.xml file:
1.0.0
...
To the new version:
1.0.1
...
After this run mvn clean install to generate a new oskari-map.war under [your-server-repository-root]/webapp-map/target. Replace the oskari-map.war under the oskari-server/webapps folder (in a setup similar to the Oskari/Jetty download package) with the new one.
3.4.3 Updating the frontend
Updating an Oskari application that is based on the sample-application template is fairly simple as well. For the frontend the oskari-frontend (and possible oskari-frontend-contrib) dependency is updated by changing the version number in package.json file from the current version in the app:
"dependencies": {
"oskari-frontend": "https://git@github.com/oskariorg/oskari-frontend.git#1.0.0"
},To the new version:
"dependencies": {
"oskari-frontend": "https://git@github.com/oskariorg/oskari-frontend.git#1.0.1"
},
When updating Oskari version, change also the application version in package.json on your application to signal that the application has been updated.
After this you need to run npm install to install any new/changed libraries and npm run build to generate a new build under [your-application-repository-root]/dist/[version on package.json]. After the build the application repository can be served as static resources from Jetty/nginx or similar server.
Finally, to use the newly built frontend you will need to update oskari-ext.properties under oskari-server/resources to link the new frontend version:
oskari.client.version=dist/[version from your package.json]Building the frontend for production
Modern web applications are developed using JavaScript syntax that is not fully supported by older browsers and the code needs to be processed before it can be used by the end-user browsers. This step also includes bundling and minifying the code so it's more compact for consumption than the human-friendly version that is used for development. This will reduce the startup time for the end-user dramatically.
With the default sample application (a pre-built version is included in the zip download) you can run npm run build to generate the minified Javascript files. This will result in folders and files generated under [your frontend repository root]/dist/.
The build works by reading a main.js file that links all the functionality/bundles you want to use in your application together and should match the appsetup (bundle collection) used on the website you are creating including any dynamic (role-based) bundles that are added on the fly. So basically all the bundles you want to use in that application. Here's an example for the basic geoportal appsetup and for an embedded map. These are processed when running npm run build.
If you take a look at the package.json script for build you can see that the parameter --env.appdef=applications is used to point the build to search for application main.js files under the applications folder. You can change this in your own app as you wish.
The best point to start customizing your app is changing the main.js file under the sample geoportal application to only include the bundles that you are using. This reduces the amount of code the end-user needs to load. You can also safely remove the 3D applications if you don't need them. The application for embedded/published maps usually stays more or less the same and the geoportal one is usually customized based on requirements.
When changing main.js to include application specific bundles and/or customizing the application you need to run npm run build again to create new minified result. You will also need to do this when updating the new version of Oskari or after changing any application specific code/bundles you want to use.
Developing the application specific code
While developing your application it's recommended to run npm run start instead of npm run build over and over again. The start script launches a Webpack dev-server for the frontend that will process the code while you develop it. The dev-server starts at port 8081 and assumes the normal backend/server is running on the default port 8080 (The oskari-server based server component). The dev-server also assumes that the client version on the server is configured in oskari-ext.properties under oskari-server/resources as:
oskari.client.version=dist/devapp3.5 Advanced configuration
The section contains advanced setup and configuration instructions which users usually do not need to worry about.
3.5.1 Advanced Jetty configuration
Jetty default configuration
The preconfigured Jetty uses these defaults. These can be changed by modifying {jetty.base}/resources/oskari-ext.properties.
Redis:
- redis running on localhost at default port (6379)
Database (Postgres with postgis extension)
- db URL: localhost in default port (5432)
- db name: oskaridb
- db user: oskari/oskari
Oskari (provided in Jetty bundle)
Custom configurations
1) Removing the unnecessary parts
Oskari-server can run with just the oskari-map webapp. If you don't need all the features, you can remove them from under {jetty.base}/webapps.
You will also need to remove the corresponding parts of the UI so users don't have access to them. This is done by removing "bundles" from "appsetups" (these are Oskari concepts: bundles provide functionalities and appsetup defines which bundles are used in your app) and currently it needs to be done by modifying the database content. Bundles are linked to appsetups in the database table portti_view_bundle_seq and functionalities are removed from the UI by deleting rows from the table.
2) Editing article content
- User guide: edit the file in {jetty.base}/resources/articlesByTag/userguide.html
- Publisher terms of use: edit the file in {jetty.base}/resources/articlesByTag/termsofuse__mappublication__en.html
3) Changing the default port**
provide port in command line:
java -jar ${jetty.home}/start.jar jetty.http.port=8080
change
{jetty.base}/resources/oskari-ext.propertieswhere ever8080is referenced
4) Proxy settings
If you need a proxy to access internet you can configure it in {jetty.base}/start.d/oskari.ini
-Dhttp.proxyHost=
-Dhttp.proxyPort=
-Dhttp.nonProxyHosts=
-Dhttps.proxyHost=
-Dhttps.proxyPort=
-Dhttps.nonProxyHosts=5) Database url/name/user/pass are changed
{jetty.base}/resources/oskari-ext.properties needs to be updated
db.url=jdbc:postgresql://[host]:[port]/[dbname]
db.username=[user]
db.password=[passwd]5) Using external Redis
{jetty.base}/resources/oskari-ext.properties needs to be updated
redis.hostname=localhost
redis.port=6379
redis.pool.size=106) How the Jetty bundle was built
See the Howto.md inside the zip-file for details
3.5.2 Setup Redis for Oskari (Optional)
Redis is used for caching data for example from statistical datasources to provide a cleaner user experience for statistical map functionalities (optional part of oskari-map webapp).
1) Get Redis
Install binaries from http://redis.io/ or from your platforms package repository.
2) Install
And startup redis-server. The default port Redis listens to is 6379.
3). Configure Oskari (Optional)
Oskari expects Redis to be found in the default port (6379) on the same server as Oskari ("localhost"). If you have it running on another host/port you need
to change the oskari-ext.properties:
redis.hostname=localhost
redis.port=63793.5.3 Configuring nginx (Optional)
This guide gives an example for configuring a reverse proxy for Oskari-server using nginx. The latest nginx version at the time of writing is 1.8.1 which has been used to test these configurations.
Using a reverse proxy is not required for development, but is recommended for production use
You can find example configurations in https://github.com/oskariorg/sample-configs/tree/master/nginx
In /etc/nginx/nginx.conf turn on gzip support
gzip on;Most of the other configurations can be done in /etc/nginx/conf.d/default.conf
Assumptions
Oskari frontend code.
Oskari frontend code should be made available in the server directory /opt/public/oskari.
This can be changed by modifying these lines:
root /opt/public/;
# Oskari frontend files
location ^~ /Oskari/ {
rewrite ^/Oskari/(.*)$ $1 break;
try_files /oskari/$1 oskari/$1/ =404;
}Oskari-server
Oskari server should be running on localhost in port 8080. This can be changed by modifying these lines:
upstream oskariserver {
server localhost:8080;
}
Cross-site request forgery protection
Java libraries don't support the SameSite-flag for cookies yet so we need to protect our service from session manipulation and CSRF by modifying the cookies on nginx. This snippet modifies the cookie Jetty gives and adds secure, httponly and samesite-flags on it. SameSite-flag means that browsers don't send for example the session cookie when requests originate from a different domain.
# Oskari-server Jetty location
location / {
...
# set all cookies to secure, httponly and samesite by modifying "path"
proxy_cookie_path / "/; secure; HttpOnly; SameSite=lax";
}HTTPS-configuration
The following enables HTTPS on the server. Add the certificates on:
/etc/nginx/ssl/public.crtfor public key/etc/nginx/ssl/private.rsafor private key
or change the configuration accordingly.
# ssl config - optional, but recommended for offering https-urls
listen 443 ssl;
ssl_certificate /etc/nginx/ssl/public.crt;
ssl_certificate_key /etc/nginx/ssl/private.rsa;
# ssl security settings - optional, but recommended
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
server_tokens off;
# /ssl configProxying requests to non-secure services
Oskari-server will proxy any map layer services that are not secure (https) to allow browsers show the map on secure domain. This requires the application server to be aware if the incoming request is secure. In Jetty this can be accomplished by forwarding some headers to Jetty hosting oskari-map:
location / {
...
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
...
}
The Jetty must also be configured to be aware that it has a reverse-proxy. This is done by modifying {jetty.base}/start.d/oskari.ini:
# ---------------------------------------
# Module: http-forwarded
# Enable X-Forwarded-For headers handling
# Needed for:
# - Webpack frontend development
# - forward proxy on production (for example nginx -> jetty) where the proxy is handling TLS etc
# ---------------------------------------
--module=http-forwardedThis enables the http-forwarded module for Jetty 9 which allows webapps like Oskari-server get additional information about the requests it receives. This allows Oskari to proxy requests that don't support HTTPS.
Note! This also adds a great amount of network traffic going through your server!
3.5.4 Server clustering (Optional)
Oskari server can be used in a clustered environment setup. A server cluster can be used to increase availability, reliability and scalability and help deal with software crashes/hardware failures etc in the system.
A simple clustered environment can be set up by having 1-n Jetty-servers (or Tomcat etc) running Oskari-server based webapp and having a load balancer like nginx as a an entrypoint that forwards requests to the Jetty-instances. The requests can be forwarded in a round-robin fashion. "Sticky sessions" are not required but could benefit debugging etc (see Identified challenges).
Currently this means:
- User sessions can be saved/tracked in Redis (shared between nodes)
- Caches communicate removals/flushes between cluster nodes
- Programmatically setting log level is communicated between cluster nodes
Note! A "cluster" with just one server instance still means persistent sessions (sessions don't "die" with server restart etc) so it's worth considering.
Enabling cluster handling
Clustered environment handling is enabled with oskari.profiles in oskari-ext.properties:
# Comma-separated list of profiles to activate.
oskari.profiles=redis-sessionAdd redis-session to the profiles list
Code examples for adding more cluster aware functionality
Check status
Check if we have cluster handling enabled:
boolean isClustered = org.oskari.cluster.ClusterManager.isClustered();Messaging
Calling ClusterManager.getClientFor([functionality id]) returns a ClusterClient instance that is created on first call and shared between returning calls. This keeps the number of Redis-connections manageable. The ClusterClient can be used to send messages and listen to messages. The ClusterManager handles that messages from the same instance are not received by listeners of that same instance. ClusterClient handles that messages from other functionalities on the same instance are not received by other functionalities for example "cache" functionality doesn't receive messages from "logger" functionality making it easier to handle messages based on the functionality.
For example cache is used by Oskari cache classes as functionality id.
Messaging also has a channel that can be used to further filter messages for specific listeners. Channel could be considered an event name for usual event based code. In the cache functionality channel is used to identify the cache instance (maplayer, dataprovider etc).
Sending messages
String functionalityId = "cache";
String channel = "maplayer";
String msg = "REM:1";
ClusterManager
.getClientFor(functionalityId)
.sendMessage(channel, msg);This would send a message REM:1 to cache functionality with channel maplayer. This would signal other cluster nodes that cache with name maplayer should be notified with the message REM:1. The functionality that is doing messaging is responsible for implementing the message parsing/protocol ie. what kind of message signifies what in the functionality.
Listening to messages
Listen to messages from other cluster nodes based on functionality id and more specific message type channel. The code below can be used to listen to messages from other cluster nodes that have been sent with the code snippet above.
String functionalityId = "cache";
String channel = "maplayer";
ClusterManager
.getClientFor(functionalityId)
.addListener(channel, (msg) -> handleClusterMsg(msg));The handleClusterMsg() will receive the REM:1 message from the above example and in the case of caches will remove the value with key 1 from the cache. Note that channel is used to identify the cache so only the maplayer cache would remove the item with key 1, not for example a cache for dataproviders.
Identified challenges
Server clusters usually present some challenges for servers environments as well. Here's something we've identified:
Logging
Logging in clustered environment means that a single page view by a single user can generate log messages on different servers (different log files). This requires an instance specific solution like:
- "sticky session" where the load balancer in front of Oskari-server forwards requests from single user to the same server instance.
- a centralized logging system where the log from each node is gathered on a central system.
The centralized logging would also benefit from a "request id" that could be used to group log messages/user without identifying the user details. This hasn't been implemented as it is not trivial and would require passing some identifier to different parts of the system that don't require it now.