Cloud Storage Manager (CSM)
1. Introduction
Cloud Storage Manager (CSM) is a document management service. It provides an easy configuration, access control, and unique API for document management regardless of storage type. Additionally, it provides a simple user interface for interaction with documents.
1.1. CSM Concepts
This section describes key concepts and terminology you need to understand to use CSM effectively.
1.1.1. Bucket concept
A bucket is a container for resources stored in CSM. Every resource is contained in a bucket.
Buckets serve several purposes:
- They organize the CSM namespace at the highest level.
- They identify the account responsible for storage.
- They play a role in access control.
1.1.2. Resource concept
Resources are the fundamental entities stored in CSM. They consist of resource data and metadata. The metadata is a set of name-value pairs that describe the resource.
Resource is uniquely identified within a bucket by a resource ID.
1.2. Architecture
CSM is a Spring boot application. It consists of the REST layer, business logic (Services), data storage (DB) and integration with external document storage.
You can see more information on diagram bellow:
We are going to dive deeper in each integration in following sections.
2. Data store
CSM needs data storage to keep information about resource usage. We support a few different types of data storage. Changing the type of data storage is done through the property bv.data.doc.store
. Current version of CSM support following types of data store:
- sql(default) - all data will be stored in sql database. Supported and tested database vendors:
- Oracle
- MS SQL
- MySQL
- DB2
- PostgreSql
- MariaDB
- mongo - use mongo db for storing data. To use mongodb all needed
spring.data.mongodb
properties must be provided.
2.1. SQL
If user starts CSM without any additional settings, H2 in-memory database will be used. To use a real database user has to provide datasource configuration, i.e. mssql:
spring:
datasource:
url: jdbc:sqlserver://localhost:1433;databaseName=test1
username: SA
password: BlueVibes(!)2019
driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
3. Buckets
This section explains how to work with buckets. We’ll explain which storage types can be used, how to configure buckets and make them secure.
3.1. Configuration
CSM aims to supports multiple Cloud providers and buckets. Information needed for connecting them should be provided in the configuration. Current implementation provides following pre-built strategies: - s3 - provides strategy for accessing s3 storage (Amazon/Swisscom or some other Amazon S3 compatible storage). In order access a bucket on s3, you should provide following configuration:
csm:
buckets:
- bucket-id: your-bucket
strategy-name: s3
settings:
endpoint:
service-endpoint: "https://xyz.swisscom.com"
#Following parameter is needed only when S3 provider have multiple regions(Amazon)
signing-region: Europe.ch
credentials:
access-key: "your_access_key"
secret-key: "your_secret_key"
#following property is optional, it should be used only in a special case when s3 bucket-name doesn't match bucket-id
s3-bucket: "target-s3-bucket"
3.2. Configuring buckets by using third party repository
There can be a case where there is no static set of buckets or buckets can be added in runtime. CSM provides a way to dynamically synchronize buckets configuration with third-party REST repository. You can enable this option through CSM configuration:
csm:
synchronization:
buckets:
enabled: true
rest:
url: http://localhost:8101/buckets
- Property
synchronization.buckets.enabled
needs to have value true in order to enable dynamic buckets configuration. - Property
synchronization.buckets.rest.url
represents endpoint of third-party API. This API endpoint needs to accept GET request with response that maps array of buckets object with same fields as configuration.
Example of buckets response:
[
{
"bucketId": "s3-bucket",
"strategyName": "s3",
"settings": {
"endpoint": {
"serviceEndpoint": "https://amazon.s3.com"
},
"credentials": {
"accessKey": "accessKey",
"secretKey": "secretKey"
}
}
}
]
If some buckets already exists in application context, they will be skipped, while rest of the buckets will be added.
Synchronization executes on application startup and then on every 15 minute in an hour (0, 15, 30, 45 minute of each hour). This way we keep all the instances of CSM synchronized in terms of their context.
3.3. Security and access control
Currently, CSM supports access control only per bucket. More granular access per resource will be supported in the future. Each bucket should have its own access control configuration. One bucket has one or more rules with allowed permissions. Therefor, for a bucket and present user, CSM will evaluate rules and aggregate permissions. Following properties represent a permission rule:
property | type | required | default | description |
---|---|---|---|---|
match-user | enum[ALL, IS_AUTHENTICATED, HAS_AUTHORITY, HAS_USER_ID] | true | Tells which method will be used for verifying is rule applicable on current user. | |
value | string | true - when matching method is HAS_AUTHORITY or HAS_USER_ID | Value used in matching process. i.e in case of matcher HAS_AUTHORITY it could be ADMIN | |
list | boolean | false | false | Represents permission for listing resource. |
download | boolean | false | false | Represents permission for downloading resource in bucket. |
upload | boolean | false | false | Represents permission for uploading resource. |
For a better understanding, let’s imagine that we have a bucket xyz
with a strategy, and following configuration:
csm:
buckets:
- bucket-id: xyz
strategy-name: astrategy
permissions:
- match-user: IS_AUTHENTICATED
list: true
download: false
upload: false
- match-user: HAS_USER_ID
value: mmustermann
list: false
download: true
upload: false
- match-user: HAS_AUTHORITY
value: ADMIN
list: false
download: false
upload: true
As we described, CSM will evaluate every rule and create permission for resource and present user. In this example for authenticated user, he will have permission to list bucket (see file names, metadata and thumbnails). Thanks to the second rule, if current user has user_id mmustermann
, he will have permission to download resource. Additionally, regarding first rule user will be authenticated, so he can list bucket content. Finally, in case that current user is mmustermann
and he has an ADMIN role, he will be able to list, download and upload a resource.
3.4 Owner of the resource
CSM recognize a need that a user is an exclusive owner of the resource. This means that only a user who is an owner can perform any operation on the resource, including being able to see it, regardless of the permission settings on the bucket. In addition, if the owner shares the resource with some other user, that user will be able to access the resource, based on the shared permissions.
3.4.1 Setting exclusive owner
There are two ways to set the exclusive owner on the resource:
- Bucket configuration: by setting
set-creator-as-owner
bucket property totrue
(default value is false). In this case, user who creates a resource automatically becomes its owner. - By calling API endpoint: Owner can change ownership on the resource by sending PUT request to
/api/v1/resources/{bucketId}/{resourceId}/owner/{ownerId}
endpoint, whereownerId
parameter is user id of the new owner. After changing an owner, old owner will not be able to access the resource.
3.4.2 Filtering resources by owner
CSM provides a REST endpoint for fetching resources for the specific owner. You can send GET request to /api/v1/resources/{bucketId}/owner/{ownerId}
, and you can get all the resources which belong to a certain user. Additionally, permission filter will be applied, so only an owner or the user with whom owner has shared the resources will be able to see teh result.
4. Resources
CSM is a simple key-value store designed to store as many resources as you want. You store these resources in one or more buckets.
Resource consists of the following:
- Resource ID – The ID that you assign to a resource. You use the resource id to retrieve the resource. It is unique per bucket.
- Content - The content which you are storing.
- Metadata - A set of name-value pairs which stores information regarding the resource.
- Links - Depending on user permission, Resource will contain 3 different links:
- Thumbnail Url - Signed download url of related thumbnail resource
- Upload Url - Signed url for uploading resource content with POST request
- Download Url - Signed url from which it’s possible to download resource content
4.1. Resource ID and Metadata
4.1.1. Resource Id
When you create a resource, CSM will create ID, which uniquely identifies the object in the bucket. If the resource already exists in a bucket (it was added manually to a storage), storage implementation will be responsible for creating resource ID.
4.1.2. Metadata
For each resource stored in a bucket, CSM maintains a set of system metadata. CSM processes this system metadata as needed. For example, CSM maintains resource creation date, creation user and size metadata and uses this information as part of resource management.
The following table provides a list of system-defined metadata:
property | description |
---|---|
resourceId | Id of resource, it is generated by system, but can be user defined |
bucketId | Id of the resource bucket. |
name | Humanly readable resource name. For example, report.pdf |
sizeInBytes | Resource size in bytes |
type | Resource content-type |
created | Resource creation date |
createdBy | User ID of a user who created a resource |
updated | Date of the last time resource was updated |
updatedBy | User ID of a lat user who updated resource |
downloadUrl | Link and metadata for downloading the resource source |
uploadUrl | Link and metadata for uploading the resource content to the storage |
thumbnailUrl | Link and metadata for accessing the resource thumbnail |
readOnly | Flag that shows if the resource is read-only |
The metadata url consist of following properties:
property | description |
---|---|
url | Url link for accessing content |
expires | The link expiration date |
headers | Custom headers that have to be included in the request. |
4.2. API Endpoints
4.2.1. Fetch resources metadata
The first step for accessing is fetching metadata of resources, for that purpose you can use one of the following endpoints:
GET /api/v1/resources/{bucketId}
- Request that lists all the resources inside the bucket. This request can be slow, especially in case of bucket with huge number of stored files.GET /api/v1/resources/{bucketId}/{resourceId}
- Request that fetches resource metadata of specific resource.
Downloading content
If a user has permission to download a certain resource, the fetched metadata for the resource will contain property downloadUrl
, otherwise, it will be null. Depends on bucket type and its configuration, the link validity can be limited by expiration date and time (property: expires
), so before you start download this property should be checked. Besides that sometimes additional headers could be required, so if they are present in metadata they should be added to the download request. Fetched URL contains the signature and changing query params or header will throw an exception.
The CSM provides two strategies for downloading resources from a bucket:
- PROXY (default) - every download request goes to the CSM. This strategy offers a higher level of security and using a link is simpler, but on the other hand, all download traffics goes over CSM.
- DIRECT - every download request goes directly to the storage provider (i.e. S3). This strategy offers faster access and less traffic on CSM service, but on the other hand it’s less secure (signed link are used) and additionally before then you use a link you should check the expiration.
The download strategy can be changed on the bucket level, by defining property downloadStrategy
.
Downloading multiple resources as zip file
CSM provides an endpoint for downloading multiple resources compressed into zip. If you want to download multiple files, you can send GET /api/v1/compressed-downloads/{bucketId}?resourceIds=resource1,resource2...
request. Query parameter resourceIds
must contain at least one resource id. User must have download
permission for all the resources that are sent in the request.
4.2.2. Upload resource
Here we distinguish two cases:
- uploading/altering an existing resource in storage
- uploading new resource Notice: user must have upload permission for the bucket
Uploading existing resource
There are two steps for uploading already existing resource:
- Fetch resource metadata - Metadata will contain pre-signed
uploadUrl
andheaders
parameters, if the user hasupload
permission on the bucket. - Upload resource content - Send
PUT
request touploadUrl
.
Uploading new resource
Upload new resource is quite similar to previous case:
-
Create new resource in csm - calling REST endpoint
POST /api/v1/resources/{bucketId}
. Request should have an object that contains resource name as thename
parameter inside the body. Result of this call is resource metadata of newly created resource. Additionally, resource can be marked as temporary, so it can be deleted after time to live expiration. This behavior can be accomplished by sendingteporary=true
andtimeToLiveInHours
parameters in the request body. -
Upload resource content - Send
PUT
request touploadUrl
.
Example of create resource request body:
{
"name": "Resource.txt"
}
Upload resource content
Once we have uploadUrl, the next step is sending. Depends on bucket type and its configuration, the link validity can be limited by expiration date and time (property: expires
), so before then you start upload this property should be checked. Besides that sometimes additional headers could be required, so if the custom headers are present in the metadata they should be added to the upload request. The URL contains the signature and changing query params or headers will throw an exception.
The CSM provides two strategies for uploading resources to a bucket:
- PROXY (default) - every upload request goes to the CSM. This strategy offers a higher level of security and using the link is simpler, but on the other hand, all upload traffics goes over CSM.
- DIRECT - every upload request goes directly to the storage provider (i.e. S3). This strategy offers faster upload and less traffic on CSM service, but on the other hand it is less secure (signed link are used) and additionally before then you use a link you should check the expiration and put custom headers.
The upload strategy can be changed on the bucket level, by defining property uploadStrategy
.
4.2.3. Delete resource
Deleting existing resource can be done by calling DELETE /api/v1/resources/{bucketId}/{resourceId}
. Resource will not be permanently deleted from the bucket, but instead it will be moved to trash folder, so it can be accessed if necessary. After deleting, resource will not be visible for CSM application, but only directly through the bucket implementation (cloud store, database etc.).
4.2.4. Mark resource as read-only
Resources in CSM can be marked as read-only by their creators. Following api is provided for that purpose:
PUT /api/v1/resources/{bucketId}/{resourceId}/readonly
set read-only flagDELETE /api/v1/resources/{bucketId}/{resourceId}/readonly
remove read-only flag
When resource is set to be read-only, no user can edit or delete the resource, regardless of the permissions. Only way to be edit resource again is to remove read-only flag by resource creator (owner).
4.2.5. Cleanup resources
CSM provides an option to configure cleanup settings, which will define which resources should be permanently deleted based on resource ID pattern and uploaded time. This functionality is used for removing temporary resources, and reducing storage size.
Resource cleanup is configured on the bucket level, like shown in example bellow:
csm:
buckets:
- bucket-id: bucket-1
strategy-name: s3
cleanup-props:
- resource-pattern: "\\.trash.*"
resource-ttl-in-days: 1
- resource-pattern: "\\.temp.*"
resource-ttl-in-days: 2
Example above means that all resources which ID starts with .trash~
or .temp~
prefix, and they have been created more than 1 and 2 days before cleanup job execution, will be deleted during cleanup.
Since the cleanup process is irreversible, it must be used carefully. This is why CSM is only supporting cleanup of hidden resources (resources which id starts with .
character). Property resource-pattern
has a validation that it has to start with \\.
sequence of characters, which means that resource id must start with .
character, in order to be deleted.
Default value of resource-ttl-in-days
property is 1 day.
CSM performs once a day, at 2:00 AM system time.
4.2.6. Update resource metadata
There is possibility to update resource metadata. Following api is provided for that purpose: PUT /api/v1/resources/{bucketId}/{resourceId}/metadata
. At this point only value for resource name can be changed.
Put request body should has following attributes:
property | type | required | description |
---|---|---|---|
name | string | false | New name for resource. |
Here is example of request body:
{
"name": "new name.docx"
}
4.3. Conversions
CSM provides an API for converting one type of resource stored in a bucket to another one. For example, you can convert resource of type word into resource of type pdf.
Multiple conversion services can be registered at the same time. Service will select an implementation based on priority and input/output type compatibility. Each converter implementation needs to provide its order, where services with higher order number have higher priority. Additionally, it has to implement canConvert
method that would tell if it supports input/output combination on resource types.
4.3.1. Adding new converter implementation
There are two ways to add new converter implementation:
- By customizing CSM service and implementing
ResourceConversionAdapter
interface - By implementing a converter which meets expected API and register it as a custom converter through configuration.
Implementation of ResourceConversionAdapter
interface
This implementation is possible in case of custom implementation of CSM service (more information in Development and Integration section). In this case, you simply add an implementation of ResourceConversionAdapter
interface and annotate it with Spring @Service
annotation.
Adding generic conversion by providing third party API
In order to add generic conversion, it’s necessary to provide API implementation that meets following criteria:
- Receives a POST request with body parameters:
resourceUrl
(full URL to resource file)inputType
(type of the resource (doc
,pdf
…))outputType
(result type of the converted resource)
- Returns a response with resource content in the body
In order to register new generic converter in CSM, you need to provide following configuration:
csm:
generic-conversions:
- service: testService # name of the service
endpoint: http://localhost:8100/convert # full URL to API conversion endpoint
order: 3 # order priority for the service
accepts:
pdf: png,doc,jpg
png: pdf
accepts
parameter receives a list of allowed output types for each allowed input type.
As the example indicates, it’s possible to add multiple instances of generic conversion APIs.
This is a preferred way to add new implementation of conversion service, since it doesn’t require new release of CSM server.
4.3.2. Converting resource to thumbnail
We have introduced three new resource types:
THUMBNAIL_SMALL
THUMBNAIL_STANDARD
THUMBNAIL_LARGE
If you call conversion service with one of the previous types as output type, it’s expected that service implementation
should convert resource to thumbnail. It’s up to the implementation itself to determine the size and image type of the converted thumbnail.
4.4. Exporting resource
CSM provides a way to convert resource and save result as a new resource. This action is called export.
REST endpoint for exporting resources is POST request with url /api/v1/exports/{bucketId}/{resourceId}
. Parameter resourceId
is the id of the resource that is stored in the bucket and should be exported.
Post request body has following attributes:
property | type | required | description |
---|---|---|---|
format | string | true | Format of the converted resource (for example: doc, pdf, png…) |
sourceFormat | string | true | Format of the original resource (for example: doc, pdf, png…) |
temporary | boolean | false | Indicates if exporting resource should be deleted after TTL period |
timeToLiveInHours | int | true | TTL in hours, applies only if resource is marked as temporary |
4.4.1. Special format types
There are several formats that don’t match resource extensions:
- THUMBNAIL_SMALL - represents small thumbnail, converter is expecting png resource
- THUMBNAIL_STANDARD - represents standard size thumbnail, converter is expecting png resource
- THUMBNAIL_LARGE - represents large size thumbnail, converter is expecting png resource
- DICOM_IMAGE - represents DICOM image format
- DICOM_ZIP - represents set of DICOM images, converter is expecting zip resource
4.4.2. Generating exported resource name
CSM provides a way for you to define the pattern for exported resource id. This pattern will be applied on all the exported resources. Default pattern is: <source_name>_<date_time>.<extension>
There are several placeholders that can be used inside the pattern: * <format>
- format of the converted resource * <extension>
- extension the converted resource * <source_format>
- format of the original resource * <source_name>
- name of the original resource * <date_time>
- current timestamp
These placeholders can be combined in order to get most descriptive resource name for exported resource. For example, if you want to have a reference to original resource inside of converted resource name.
Pattern is configured on bucket level, as parameter export-name-pattern
, for example:
csm:
buckets:
- bucket-id: your-bucket
strategy-name: s3
export-name-pattern: "<source_name>.<extension>"
Prefix for exported resources
If you want to group all exported resources, you can do that with the prefix in their resource name. It’s possible to configure the wanted prefix, that will be added to the beginning of resource id.
Prefix is configured on bucket level, as parameter export-prefix
, for example:
csm:
buckets:
- bucket-id: your-bucket
strategy-name: s3
export-name-pattern: "<source_name>.<extension>"
export-prefix: "converted-"
Prefix can be an empty string or a string that matches criteria: Alphanumeric characters ([a-zA-Z0-9]) or hyphens (-). Default value is export-of-
.
4.4.3. Setting Time to Live for exported resource
In case that exported resource should exist only for a specific period of time, it can be marked as temporary. It the exported resource is marked as temporary, it will be deleted after time to live expiration. Cleanup job is performed once a day.
4.5. Creating thumbnail for resource
One way of presenting a preview of resource is a thumbnail image. CSM resource metadata contains a link to related thumbnail if it exists.
In current implementation there are two ways of creating a thumbnail for specific resource:
- Lazy creation - at the moment when resource is fetch from the service related thumbnail will be created if it doesn’t exist. You can enable this feature on the bucket level through configuration property
create-thumbnail-on-get-resource: true
. By default, this feature is disabled. Configuration example and diagram are shown bellow:
csm:
buckets:
- bucket-id: xyz
strategy-name: astrategy
create-thumbnail-on-get-resource: true
- Explicit creation of thumbnail through POST request: REST endpoint
/api/v1/thumbnails/{bucketId}/{resourceId}
. The result of this request is an object that contains signed URL to created thumbnail. Diagram of explicit creation is shown bellow:
A prerequisite for creation of thumbnail is existence of suitable conversion service implementation that is capable of converting specific resource type to thumbnail.
In a case of uploading new content to existing resource, thumbnail will not be automatically created. Instead, it should be created through REST endpoint.
Creating thumbnail will use conversion implementation, so necessary conversion implementation should be provided. This implementation should accept THUMBNAIL_SMALL, THUMBNAIL_STANDARD and THUMBNAIL_LARGE as output format parameters.
4.6. Actions
If you need to execute an action, or a chain of multiple actions over a resource, CSM provides an API for that purpose. It is possible to use built-in actions, as well as configuring a generic action by providing connection to third party API.
You can execute actions by sending a POST request to /api/v1/actions/{bucketId}/{resourceId}
endpoint. Both bucketId
and resourceId
are required. Additionally, request body should contain an array of action objects, where every object has 2 parameters:
actionId
- id of an action that is going to be executed. Each action will take a result of previous action as a resource content.payload
- a map of label-value parameters that are going to be forwarded to each action. These parameters represent all the necessary parameters for actions execution. Parameters likebucketId
,resourceId
andresourceUrl
are added to payload by CSM service.
Actions will be executed in order in which they are sent in body request array.
Result of each action is a resource content.
Final action result will be saved as resource content in CSM.
You can see a diagram of action chane execution bellow:
You can see all actions performed on a resource by sending a GET request to /api/v1/actions/{bucketId}/{resourceId}/history
. For every action performed you can see the user who performed it and the timestamp when it was performed.
4.6.1. Built-in actions
Built-in actions: - convert
- converts resource from one file type to another (e.g. RTF do DOCX, DOCX to PDF…) and returns result resource. Required payload parameters for this action are resourceUrl
, sourceFormat
and format
.
4.6.2. Generic actions with third party API
It you want to implement custom actions, you should provide a custom API implementation. This API should meet following criteria: * Endpoint for POST request that consumes multipart/form-data, with 2 parameters: * payload
- json object with label-value parameters * resource
- resource file on which action will be performed
API endpoint should return a stream that contains result of an action.
4.6.3. Configuring generic action
Adding a generic action is strait forward. You just needs to add 2 parameters: - actionId - (this action id will be sent in action chain) - URL to third party API endpoint
Example:
csm:
actions:
- action-id: populate-images
endpoint: http://localhost:8100/api/populate-images
- action-id: populate-labels
endpoint: http://localhost:8111/api/populate-labels
4.7. Sharing resource
You can share the resources that you store in CSM with anyone.
When you share from CSM, you can control whether people can edit, delete, or only view the file.
You can see share resource execution on diagram bellow:
As shown on a diagram, you can provide external user with access link. After accessing this link, external user will be redirected to page, where he can see all the resources shared with him. Depending on the access right, user will be able to perform actions on these resources.
At this moment, only download action iz implemented.
You can see external user interaction with the shared link on diagram bellow:
Note: in the future, external user will have to authenticate before being redirected to UI. Authentication implementation is yet to be decided.
4.7.1. Share API Endpoints
Share resource
You can share the resource by sending POST /api/v1/shares/{bucketId}/{resourceId}
request. Post request body has following attributes:
property | type | required | description |
---|---|---|---|
sharedWith | string | true | ID od u user your sharing resource with |
expirationInMinutes | long | no | Optional time to live in minutes for created share. |
resourcePermission.list | string | true | Will user be able to access shared file |
resourcePermission.download | string | true | Will user be able to download shared file |
resourcePermission.delete | string | true | Will user be able to delete shared file |
resourcePermission.upload | string | true | Will user be able to upload shared file |
Here is example ob request body:
{
"sharedWith": "guestUser",
"expirationInMinutes": 1000,
"resourcePermission": {
"list": true,
"download": true,
"delete": false,
"upload": false
}
}
Response will contain the similar object, with some additional data, including access link:
{
"resourceId": "Document_test.docx",
"bucketId": "bucket-1",
"sharedBy": "you",
"sharedWith": "guestUser",
"created": 1595405662067,
"accessLink": "http://localhost/csm/shares/6bd9cfaf-691e-4c89-8751-b4880096a827",
"resourcePermission": {
"list": true,
"download": true,
"upload": false,
"delete": true
}
}
Value of accessLink
response parameter can be sent to user. User will be able to access shared document trough provided link.
List shared resources
You can list all the shared resources by sending GET /api/v1/shares/{bucketId}
request. Response will list all the share
object that current user has permissions to see. CSM resolve permissions based on buckets permissions (see buckets security section) and permissions on the shared resource itself.
For example, if user has role ADMIN, and ADMIN role can list all resources on the bucket, user will be able to list all the shared resources. Otherwise, if a user doesn’t have any permission on bucket level, he will be able to list only resources shared with him.
This request can be further filtered u adding sharedWith
query parameter. This parameter can have two different values: * userId - Id of a user who shared resources we want to list * “me” - Service will list shared resources of the current user
List all shares for a resource
If you want to check all the user that a specific resource has been shared with, you can do it by sending GET /api/v1/shares/{bucketId}/{resourceId}
request.
Same as in the previous section, user will be able to see shares only if has permission to list provided resource.
Update existing share
If you want to update ean existing share (for example, update permissions), you can do it by sending POST /api/v1/shares/{bucketId}/{resourceId}
request. Request body and response have the same structure as in create request. Additional notice, sharedWith
parameter must be the same as in existing share, or service will return an error.
Only the user with update
permission on the resource can update the share.
Delete share
If you want to revoke resource share, you can do it by sending DELETE /api/v1/shares/{bucketId}/{resourceId}/{userId}
request, where userId
URL parameter should match sharedWith
parameter of the share you want to delete.
Only the user with delete
permission on the resource can delete the share.
Delete all shares for user
If you want to revoke all shares, you can do it by sending DELETE /api/v1/shares/{bucketId}/{userId}
request, where userId
URL parameter should match sharedWith
parameter of the share you want to delete.
This will mark all the shares with provided user as inactive. Additionally, it will mark shae link as inactive as well.
4.7.2. Shared link API Endpoints
Fetch shared link by id
If you want to fetch a shared link, you can do it by sending GET /api/v1/shared-link/{sharedLinkId}
request.
Response will contain json object with link data:
{
"sharedLinkId": "link-id",
"bucketId": "bucket-id",
"userId": "user-id",
"created": "2021-06-11T14:33:46.583+00:00",
"validUntil": "2021-06-21T14:33:46.583+00:00"
}
property | type | required | example |
---|---|---|---|
sharedLinkId | string | true | link123 |
bucketId | string | true | bucket456 |
userId | string | true | user789 |
created | date | false | 2021-06-11T14:33:46.583+00:00 |
validUntil | date | false | 2021-06-21T14:33:46.583+00:00 |
4.7.3. Configuration
Regarding resource sharing, there are several configuration properties that you can adjust:
property | type | default value | required | description |
---|---|---|---|---|
csm.public-url | string | http://localhost:14000/csm | true | Public URL of CSM including context path. This should be set for each environment differently |
sharedLinkExpirationInMinutes | long | 60 * 24 * 5 (five days) | true | Expiration time of share link in minutes. This is a bucket level parameter! |
sharedLinkExpirationInMinutesAfterAccessed | long | 60 (1 hour) | true | Expiration time of share link in minutes after it’s been accessed . This is a bucket level parameter |
4.7.4. Security
As already shown in create share section, you should provide permission for user on shared resource. There are four permissions to be set:
"resourcePermission": {
"list": true,
"download": true,
"upload": false,
"delete": true
}
These permissions will be aggregated with bucket level permissions (see section 3.3 Security and access control).
4.7.5 Opening shared resource
For user to open shared resource couple of things have to be set up. There must be instance of CSM inside a page. csm.csm-page-url
must be provided with correct URL to the page containing CSM, for example: csm.csm-page-url: "https://host.docker.internal/bv/medical/#/workspace/csm-demo/page/csm"
. Also, the param supportOpenDocument
must be provided to CSM micro application instance, and it’s value set to true
. For example:
{
"instanceId": "bv-csm",
"name": "CSM",
"microAppId": "csm",
"componentId": "bv-csm",
"params": [
{
"name": "supportOpenDocument",
"value": "true"
}
]
}
The shared resource is opened in read-only mode.
4.8. Resource editing management
CSM supports tracking of the resources currently being edited (e.g. excel file). Document can only be edited by one user at the time.
When user starts document editing, document is locked for further editing until user finishes his editing session. Lock is represented by the entry in resource editing status table. If the entry is present, document is locked. Resources editing status data is kept in the database. In current version of the CSM following data is stored: - Resource and bucket id of the document. - Resource key which identifies the version of the resource currently edited. - Datetime of the last document update.
To set the editing status, editor app must manage following REST endpoint /api/v1/resource-editing-status
. Following operations are currently available: - Start editing: Creates the status, generates the resource key and locks further editing. - End editing: Removes the editing status of the resource and unlocks the resource for editing. - Get currently edited resource with resource key: Retrieves the metadata of the resource currently being edited. This call is used by editors to store updated version in data store. For that reason updated
datetime value for status is also updated. - Get edited resource status: Return current status of the edited resource. Used by editors to check the status of the document lock. More details about this process are below.
When editor locks the document, lock is granted for specific period of time. If editor is idle (no updates arriving), document lock can expire. This is a safety mechanism to prevent infinite lock of the document if for example user forgets to close the editor. This time can be configured by following property:
csm:
editing-lock-timeout-in-minutes: 5
4.9. Recording activity logs
CSM will log most of the activities on the resources and other entities, like following:
- Downloading
- Converting
- Exporting
- Access to shared resources
- Custom action execution
Activity log will store meta information like resource id, bucket id, action, user id, ip address and timestamp when the log is created.
4.9.1. Accessing activity logs
Csm provides a REST api for accessing activity logs. You can get all the logs by sending GET /api/v1/activity-logs/{bucketId}?entityId={resourceId}&userId={userId}
request. Both entityId
and userId
query parameters are optional. They serve for filtering logs.
You can see response example bellow:
{
"resources": [
{
"activityLogId": "57d82882-c66c-40a8-8691-64ee73613b81",
"userId": "testUser",
"bucketId": "bucket-1",
"entityId": "96ce41ef-c5bb-4125-8045-ece15537620d",
"entityType": "Share",
"activity": "Resource Document_test.docx has been shared with user guestUser",
"ipAddress": "172.20.0.1",
"created": 1597743040125
},
{
"activityLogId": "0120d151-8d43-4aaf-912c-427b2ffc36a5",
"userId": "testUser",
"bucketId": "bucket-1",
"entityId": "96ce41ef-c5bb-4125-8045-ece15537620d",
"entityType": "Share",
"activity": "Resource Document_test.docx has been seen by user guestUser",
"ipAddress": "172.20.0.1",
"created": 1597743048932
}
]
}
4.9.1. Logging a custom activity
If you want to create your own activity record, CSM provides a REST API endpoint for that. You need to send GET /api/v1/resource-activities/{bucketId}
request with following body properties:
property | type | required | description |
---|---|---|---|
entityId | string | true | Id of the entity which log you want to save |
entityType | string | true | Type of the entity which log you want to save. For example, Resource, Share… |
activity | String | true | Activity description you want to log |
Additional properties like user_id, timestamp and ip address will be provided by CSM.
5. Development and Integration
5.1. API documentation
After starting CSM, swagger documentation is available on http://{your-domain:port}/swagger-ui.html
5.2. Running a demo
There are two ways to run demo/mock csm service: * standalone jar
```
java -jar bv-csm-service.jar -Dspring.profiles.include=demo
```
- docker
docker pull docker.corp.ametiq.com/bv-csm:0.1.0 docker run -d -i -p "14000:14000" --env SPRING_PROFILES_INCLUDE=demo --name bv-csm-service docker.corp.ametiq.com/bv-csm:1.0.0
5.3. Building custom CSM or use prebuilt service
As it is mentioned earlier, CSM tends to support most popular cloud storage vendors. Therefor, we basically decided to provided pre built service as jar or docker image. This way, user can just configure service and start to use application service. If there is need for some customizations (storage is not supported, custom access policy, etc) developer can use csm-core library and by defining a few beans in order to customize service.
5.3.2. Building own customized CSM service
The first step that should be done is to include dependency in application pom:
<dependency>
<groupId>io.bluevibes</groupId>
<artifactId>bv-csm</artifactId>
</dependency>
The second and final step for enabling CSM in you application is annotation Application
or a configuration with @EnableCsm
.
@EnableCsm
@SpringBootApplication
public class Application {
//...
}
5.3.1. Providing custom strategy for managing storage
To provide a custom strategy, you need to extend abstract class DefaultBucketStrategyFactory
or in a special case even implement interface BucketStrategyFactory
. Beside, depending on what behavior you want to change, you need to implement certain strategy bean (AccessPolicy
, DownloadProxy
or ResourceAdapter
). In case that a custom strategy needs an additional bucket settings, then property class should be created in a scanned package and annotated with @PerBucketProperties
.
For better understanding, in following diagram we assumed that we have one standard predefined strategy and one custom implemented by a developer.
CSM initialization and creating bucket’s contexts:
Handling request for getting a resource: