Media location and URL inconsistency when using Amazon S3

[gazal]

We're trying to use Amazon S3 for all of our hybris media. Not just some new media folders.

The config we're using:

media.default.storage.strategy=s3MediaStorageStrategy
media.globalSettings.s3MediaStorageStrategy.bucketId=hybris-media-7890

This along with the right reverse proxy settings seems to work find to render media on the storefront web pages correctly.

But we noticed some inconsistency in the way media is stored.

Example media object created in hybris with LocalFileMediaStorageStrategy:

{
    "code": "base_catalog_ui_config",
    "location": "h33/h8a/8861676830750.xml",
    "url": "/medias/sys_master/h33/h8a/8861676830750.xml"
}

Example media object created in hybris with S3MediaStorageStrategy:

{
    "code": "editor_jasperMediaconfig",
    "location": "sys_master/root/h7a/h86/12249255575582/Editor_JasperMedia.xml",
    "url": "/medias/sys_master/sys_master/root/h7a/h86/12249255575582/Editor_JasperMedia.xml"
}

We actually extended S3MediaStorageStrategy to make the prefix consistent without which it would've been:

"location": "sys-master/root/h7a/h86/12249255575582/Editor_JasperMedia.xml"
"url": "/medias/sys_master/sys-master/root/h7a/h86/12249255575582/Editor_JasperMedia.xml"

For the media originally created using LocalFileMediaStrategy, we're getting errors like:

2020-01-16 14:59:53 [ERROR|||de.hybris.platform.cockpit.services.config.impl.DefaultUIConfigurationService]  Cannot create ui component configuration with code base: de.hybris.platform.media.exceptions.MediaNotFoundException: Media not found (requested media location: h37/h8b/8861685350430.xml)
de.hybris.platform.servicelayer.media.NoDataAvailableException: de.hybris.platform.media.exceptions.MediaNotFoundException: Media not found (requested media location: h37/h8b/8861685350430.xml)
	at de.hybris.platform.servicelayer.media.impl.DefaultMediaService.getStreamFromMedia(DefaultMediaService.java:274) ~[coreserver.jar:?]
	at de.hybris.platform.cockpit.services.config.impl.DefaultUIConfigurationService.createComponentConfiguration(DefaultUIConfigurationService.java:447) [cockpitserver.jar:?]
	at de.hybris.platform.cockpit.services.config.impl.DefaultUIConfigurationService.getComponentConfiguration(DefaultUIConfigurationService.java:288) [cockpitserver.jar:?]
	at de.hybris.platform.cockpit.services.config.impl.DefaultUIConfigurationService.getComponentConfiguration(DefaultUIConfigurationService.java:356) [cockpitserver.jar:?]
	at de.hybris.platform.cockpit.services.label.impl.DefaultLabelService.getProvider(DefaultLabelService.java:157) [cockpitserver.jar:?]
	at de.hybris.platform.cockpit.services.label.impl.DefaultLabelService.access$0(DefaultLabelService.java:153) [cockpitserver.jar:?]
	at de.hybris.platform.cockpit.services.label.impl.DefaultLabelService$1.execute(DefaultLabelService.java:67) [cockpitserver.jar:?]
	at de.hybris.platform.servicelayer.session.impl.DefaultSessionService.executeInLocalView(DefaultSessionService.java:89) [coreserver.jar:?]
	at de.hybris.platform.cockpit.services.label.impl.DefaultLabelService.getObjectTextLabelForTypedObject(DefaultLabelService.java:56) [cockpitserver.jar:?]
	at de.hybris.platform.cockpit.services.label.impl.DefaultLabelService.getObjectTextLabel(DefaultLabelService.java:46) [cockpitserver.jar:?]
	at de.hybris.platform.cockpit.components.navigationarea.renderer.AbstracttSectionSelectorSectionListRenderer.render(AbstracttSectionSelectorSectionListRenderer.java:124) [cockpitserver.jar:?]
	at org.zkoss.zul.Listbox$Renderer.render(Listbox.java:2336) [zul-3.6.4-hybris-xss-patched.jar:3.6.4]
	at org.zkoss.zul.Listbox.onInitRender(Listbox.java:2226) [zul-3.6.4-hybris-xss-patched.jar:3.6.4]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_232]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_232]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]
	at org.zkoss.zk.ui.impl.EventProcessor.process0(EventProcessor.java:209) [zk-3.6.4-hybris-patched.jar:3.6.4]
	at org.zkoss.zk.ui.impl.EventProcessor.process(EventProcessor.java:141) [zk-3.6.4-hybris-patched.jar:3.6.4]
	at org.zkoss.zk.ui.impl.EventProcessingThreadImpl.process0(EventProcessingThreadImpl.java:519) [zk-3.6.4-hybris-patched.jar:3.6.4]
	at org.zkoss.zk.ui.impl.EventProcessingThreadImpl.run(EventProcessingThreadImpl.java:446) [zk-3.6.4-hybris-patched.jar:3.6.4]
Caused by: de.hybris.platform.media.exceptions.MediaNotFoundException: Media not found (requested media location: h37/h8b/8861685350430.xml)
	at de.hybris.platform.amazon.media.storage.S3MediaStorageStrategy.getSize(S3MediaStorageStrategy.java:261) ~[amazoncloudserver.jar:?]
	at de.hybris.platform.jalo.media.MediaManager$1.getSize(MediaManager.java:161) ~[coreserver.jar:?]
	at de.hybris.platform.media.storage.impl.DefaultLocalMediaFileCacheService.isStreamBiggerThanCacheSize(DefaultLocalMediaFileCacheService.java:215) ~[coreserver.jar:?]
	at de.hybris.platform.media.storage.impl.DefaultLocalMediaFileCacheService.getMediaCacheStream(DefaultLocalMediaFileCacheService.java:192) ~[coreserver.jar:?]
	at de.hybris.platform.media.storage.impl.DefaultLocalMediaFileCacheService.storeOrGetAsStream(DefaultLocalMediaFileCacheService.java:178) ~[coreserver.jar:?]
	at de.hybris.platform.jalo.media.MediaManager.getMediaAsStream(MediaManager.java:366) ~[coreserver.jar:?]
	at de.hybris.platform.jalo.media.MediaManager.getMediaAsStream(MediaManager.java:386) ~[coreserver.jar:?]
	at de.hybris.platform.servicelayer.media.impl.DefaultMediaService.getStreamFromMedia(DefaultMediaService.java:270) ~[coreserver.jar:?]
	... 20 more

The file is actually present in the s3 bucket, but with path; sys_master/h37/h8b/8861685350430.xml

We also check how they are stored in DB, here are the relevant columns:

| PK | p_datapk | p_location | p_realfilename | p_code | p_internalurl | p_folder | p_subfolderpath |

| 8796130672670 | 8861685350430 | NULL | Base_CMSSite.xml | base_cmssite_ui_config | replicated273654712 | 8796093055030 | h37/h8b/ |

| 8796094988318 | 12249255575582 | sys_master/root/h7a/h86/12249255575582/Editor_JasperMedia.xml | Editor_JasperMedia.xml | editor_jasperMediaconfig | replicated273654712 | 8796093055030 | ha5/hb8/ |