Change in image folder storage strategy from default to S3 is causing deletion of media files in sys_master folder?

[Former Member]

We have a unique situation. We are trying to use S3 storage strategy for images folder. Previously this folder was using default strategy (standard config). Our S3 config looks something like following,

 media.folder.images.storage.strategy=s3MediaStorageStrategy
 media.folder.images.url.strategy=s3MediaURLStrategy
 media.folder.images.accessKeyId=OURKEY
 media.folder.images.secretAccessKey=OURSECRET
 media.folder.images.endpoint=s3.amazonaws.com
 media.folder.images.bucketId=ourbucket
 media.folder.images.url.signed=false
 media.folder.images.url.unsigned.https=true
 media.folder.images.url.unsigned.virtualHost=false
 media.folder.images.url.validFor=120

In our development environment we are using DB dump and copy of sys_master folder from production (exluding images). We have copied all content of images folder to S3 bucket.

Now interesting bit, when we start Hybris locally it is deleting all the files in sys_master folder although it is keeping the folder structure as it is.

We tried several times with variety of configs but still same outcomes.

What we are missing here?

Our Hybris version: 5.0.2.1

This is definitely not an Amazon S3 issue. Our local.properties and project.properties are same in Prod and Dev with minor config differences.

Update: We enabled the media debug logging log4j.logger.de.hybris.platform.media = debug and this is what we found, clearly Hybris is removing the actually file assuming the cached file.

 INFO   | jvm 1    | main    | 2014/09/16 14:48:20.944 | DEBUG [Cluster 0: Lucenesearch-RebuildIndex-Job::de.hybris.platform.lucenesearch.jalo.RebuildIndexJob] (Cluster 0: Lucenesearch-RebuildIndex-Job) [DefaultLocalMediaFileCacheService$MediaCacheUnit] Removed cached file: C:\Work\hybris\data\media\sys_master\hfc\ha2\8831546982430.gif​

This is not an issue when either we use only default storage strategy or only s3 storage storage strategy, only when we use both we see this happening.

[Former Member]

Hi ,

I am also facing the same issue, can you please let me how this issue was fixed.

[Former Member]

Hi all,

Is there a fix for this issue?

We are using local and amazonS3 strategies and every second time when image is imported it is deleted from the amazonS3 storage.. but appears as application/octet stream (incorrect image and mime type) on hybris medias..

[Former Member]

Hello Lynna,

we had the same problem. The hint in Your message helped a lot, thank you. Now I might have a solution to this problem.

With your logging setting, you can see the recreation of the media cache using the files in the configured cache folders. These messages look like this:

 DEBUG [] () [MediaCacheRecreator] >>> Recreating media cache region using existing cached files [cache folder: ...\hybris\hybris\data\media\sys_master\cache]
 DEBUG [] () [MediaCacheRecreator] Added new cache entry MediaCacheKey [tenantId=master, location=123, cacheFolder=cache] MediaCacheUnit [cachedFile=...\hybris\data\media\sys_master\cache\..., markedAsEvicted=false, usageCounter=0]
 ...
 DEBUG [] () [MediaCacheRecreator] >>> Media cache region recreated successfully [number of processed files: 1234]
 DEBUG [] () [MediaCacheRecreator] >>> Recreating media cache region using existing cached files [cache folder: ...\hybris\hybris\data\media\sys_master\]
 DEBUG [] () [MediaCacheRecreator] Added new cache entry MediaCacheKey [tenantId=master, location=123, cacheFolder=null] MediaCacheUnit [cachedFile=...\hybris\data\media\sys_master\..., markedAsEvicted=false, usageCounter=0]
 ...
 DEBUG [] () [MediaCacheRecreator] >>> Media cache region recreated successfully [number of processed files: 5678]

I recognized, that these messages appear twice.

1. The first time the initial message has this path "...\hybris\hybris\data\media\sys_master\cache" and the cacheFolder in the following messages is set to cache.

2. The second time the initial message has this path "...\hybris\hybris\data\media\sys_master\" and the cacheFolder in the following messages is set to null.

The second group of messages got me thinking. Why is every entry in the sys-master folder considered as a cache file? And if so, are the entries considered as a chached item? Which in turn would explain, why they are deleted.

Then I figured I have to set a cache folder for my media strategy. Since I have set the following property, the second group of messages do not show up anymore. So I think they are not considered as cache itmes anymore.

 media.globalSettings.myStorageStrategy.local.cache.rootCacheFolder=my-cache

Yet I could not verify that medias won't be deleted again, but logs look fine.

Happy debugging,

Johannes

[Former Member]

Hi,

We had the same problem and setting a cache folder on our custom media strategy fixed the problem.

Thanks,

Ilie

[Former Member]

Hello,

In order to move media between folders, you should use one of the recommended mechanisms provided by hybris.

Any external storage system uses a local cache to improve performance, as explained in the documentation. Depending on your configurations, that logs are normal.

My question is...If you try to access the item that image is used for...Does the image appear?

In order to force this, you can try assigning to a product the image evicted from the cache, in your log 8831546982430.gif​ and then loading the product.

Best regards,