Media Replication Directories on Hybris cluster nodes.

[Former Member]

Hi All, I have one question regarding clustering, precisely "Media Replication Directories". I followed the wiki page: https://wiki.hybris.com/display/release5/Node-specific+Configurations and my nodes configuration for that look like:

local.properties node 1

cluster.id=0 cluster.0.media.read.dir=/var/www/hybris_app/media1 cluster.1.media.read.dir=/var/www/hybris_app/media2 cluster.0.media.replication.dirs=/var/www/hybris_app/media1;/var/www/hybris_app/media2

local.properties node 2

cluster.id=1 cluster.0.media.read.dir=/var/www/hybris_app/media1 cluster.1.media.read.dir=/var/www/hybris_app/media2 cluster.1.media.replication.dirs=/var/www/hybris_app/media1;/var/www/hybris_app/media2

On remote server node1 I manually created /media1, on the other server node2 I manually created /media2. The result of this configuration is than on those 2 nodes I have 2 catalogs: media1 and media2. When I use HMC on node 1 tu upload some sample file it is created in the same path in /media1 and media2. But it is not created on there's drive. The same is when using HMC on node 2: the file is created physically on media1 and media 2 (on node's2 drive). But it's not created on node's 1 drive :( What I can do is I can look for this sample files using node's 2 HMC, and I see it as they're on the same DB. But why the sample files are not created physically on each node? Maybe I don't understand the idea of media replication. But if anyone has an idea, or followed the same problem I would be more than greatfull to hear the answer.

[Former Member · Accepted solution]

Hi Adriana,

For the configuration of the media files in the cluster mode, we see three possibilities which are as follows;

1. Using the replication.dir and configuration with enabling every cluster node to write the media to the media folder of every node For example, if you specify; media.replication.dirs=C:\media;N:\remotedir media (Backslashes need to be escaped via another backslash.) where N:\remotedir\media is a directory on a remote server, then all media created will be saved on the local hard disk (C:\media) and on the remote server (N:\remotedir\media).
2. Using a shared driver e.g. SAN to store the medias and mount this to the data directory of every node (NFS also possible in our eyes the most performant varaity) Place from where the media filter (/medias/) is reading the media files. As path you can use something like '/usr/var/media' or 'c:\media' or '\\hostname\shared_folder' media.read.dir=${HYBRIS_DATA_DIR}/media
3. using some sync software what replicates the medias binary files to every node's media directory

All three solutions are used by customers successfully

[Former Member]

Another option would be to use MongoDB Gridfs as a storage. We have an extension called gridfsstorage which handles it.

[former_member602476]

Hi Piotr, that is a good option to have, especially in cloud environments. Will this extension handle any type of media or just product and content related media in images folder ?For example will it handle any out of the box hybris media such as cronjob, impex, email-body, catalogsync media?

[Former Member]

I think a wiki page is called-for. The general gist is available on StackOverflow

http://stackoverflow.com/questions/17425153/distributed-file-systems-gridfs-vs-glusterfs-vs-ceph-vs-...

A benefit (or drawback?) of the MongoDB GridFS strategy is that it doesn't expose media files to the sysadmin. If GridFS is available, then the stuff is just there, with less risk of accidentally deleting some media files.

[Former Member]

Hi,

It acts as a standard media storage, so yes, it will handle all your medias. What's more - it is MongoDB, so you can leverage all the goodies as scaling by adding new nodes to the Mongo etc.

[Former Member]

NFS is not the only filesystem sharing option. If there are a few hybris cluster nodes, maybe you should consider using a distributed filesystem like Ceph or Gluster (probably others as well) in order to get additional benefits like fault tolerance and point-in-time snapshots while providing the single view of media files across the entire cluster required for hybris clusters.

http://docs.ceph.com/docs/master/cephfs/

http://www.gluster.org/community/documentation/index.php/GlusterFS_Concepts

[Former Member]

Hi All, yes - I set my media replication dirs as NFS shared discs spaces and cross mounted it on two nodes and it works now. Thank you.

[Former Member]

In a cluster environment, media replication or a shared folder must be set to have same resource being accessed at all nodes

[former_member602476]

If NFS is not an option, and your goal is to share media between nodes there is another option, which I don't really prefer, but could work as a last resort. You could use rsync to push media from a back-office node to the rest of the nodes. However you'd have to be careful, rsync is one way, basically the activities that would require media uploads would have to be done on the specific node that does the rsync to the rest of the nodes. So basically depending on your use cases, you'd pin your PCM, WCMS, .... users and any automated jobs that run media imports to the node the rsync is done from. I've seen customers do this and run rsync every 5 minutes. The problem with this approach is that you'd have a 5 minute lag between the when media is published and when it's available on the other nodes. Another syncing strategy is using lsyncd which is real time sync, but I have not seen that used with hybris as of now.

[Former Member]

Hi Eoin, I'm trying to find a good solution cause my Client's admins don't allow me to use NFS on their servers.

[Former Member]

Hi Adriana,

I don't think you can have replication or even a cluster without some form of disk sharing. You either share a single disk mounted on all nodes (a la Cristian's suggestion), or you cross-mount disks and replicate...

[former_member602476]

Hi Adriana, I think you are expecting node1 to write on node2's filesystem but there is no nfs sharing configured in your case. In the examples on the wiki(windows examples), the media replication directories are shares. Basically if you want node1 to write on node2's folder you would need to create a share on node2, and mount that on node1 (if you are running a linux os) . Then your replication dir should point to that mount point. I am not sure what you are trying to accomplish by using node specific properties. Most projects needs are to share the same media folder between all clusters; in that case you'd create an nfs share, mount that to all of your nodes using the same mount point and set the media.read.dirs and media.replication.dirs to the mount point. No need to mess with node specific settings: For example, assuming /my/mediafolder was configured to point to your nfs share on each server, then you could set the following in local.properties on each node:

media.read.dir=/my/mediafolder media.replication.dirs=/my/mediafolder

I hope this helps.

[Former Member]

Hi Cristian,

I think what you're describing is not replication. You have a single point of failure in your shared disk and you have network traffic/reads for every media system access which I'd expect to be less performant.

Assuming: Node1:

• /var/www/hybris_app/media1 = local

• /var/www/hybris_app/media2 = remotely mounted from node2

Node2:

• /var/www/hybris_app/media1 = remotely mounted from node1

• /var/www/hybris_app/media2 = local

Then I'd expect Adriana's configuration to work (based on the docs).

Adriana, are you cross mounting the disks like that?

[Former Member]

Hi Eoin,

short question: what will happen when Node2 fails and Node1 creates a file in meantime and AFTER that Node2 Comes up, will it receive the lost file somehow?

Best Regards